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ABSTRACT 



A study was conducted to determine how the Career 
Preparation Assessment portfolio (CPA) , tailored to be used in local school 
reform efforts, was implemented in 10 schools in 1998-99. The research was 
organized and reported using these four clusters of research questions : 
tailoring the CPA, implementation, student performance and assessment, and 
whole-school change. The study found that: (1) the tailoring process enabled 

schools to customize the CPA to their own needs and increase ownership; (2) 
the tailored CPA can be implemented successfully in schools possessing 
certain structural characteristics generally associated with schools that are 
engaged in a whole- school reform; (3) reliable scoring of the CPA requires a 
commitment to conducting scoring with substantial training; (4) student 
performance on the tailored CPA was acceptable to implementing teachers; and 
(5) a strong majority of project teachers believe the tailored CPA is a 
valuable teaching tool that promotes changes in classroom practice and can 
support school -wide reform. (Appendices include both original and tailored 
CPA standards and rubrics, and student, teacher and scorer survey 
instruments.) (KC) 
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EXECUTIVE SUMMARY 



Introduction 

In 1994, WestEd embarked on an effort to develop a scalable, reliable, performance-based 
assessment that supported systemic reform (particularly reform supported by School-to-Career 
principles) and improved student achievement. WestEd created a standards-based portfolio of 
well-defined student tasks called the Career Preparation Assessment portfolio, or CPA, variations 
of which have been implemented and tested in schools since the 1994-95 school year. 

Over the first three years of the CPA’s implementation, WestEd saw that the fixed-model 
approach hindered the portfolio’s ability to be accepted by entire faculties and to be integrated 
into local reform efforts. High school faculties wanted the ability to alter the portfolio’s tasks 
and standards. In response, in the 1998-1999 school year, WestEd developed a model for 
tailoring the CPA’s standards and components to meet local needs. 

The power of the tailored CPA is fourfold: 

1. The tailoring process and use of the portfolio improves teachers’ expertise in and 
commitment to teaching and assessing standards. 

2. The portfolio’s interdisciplinary nature creates the basis for comparison of student work and 
tasks across a school’s curriculum, thereby providing a vehicle for all teachers to look at 
students in terms of school-wide, versus a single discipline’s, student outcomes. 
Concomitantly, the portfolio’s interdisciplinary nature encourages teachers to collaborate 
across disciplines so that students attain those school-wide outcomes. 

3. Unlike many school-wide assessments (e.g., standardized tests), teachers play a major role in 
selecting and constructing this assessment. The teacher role increases the likelihood that 
they will use performance data from across the curriculum to inform their teaching. 

4. Because the value of the portfolio as a whole is greater than the sum of its parts and because 
it focuses on skills needed for life after high school, students become more motivated to 
achieve the portfolio’s standards. 

Study of, and support for, the CPA aligns closely with WestEd’ s approach to its national 
specialty area of assessment — improving student performance and bolstering whole-school 
change through focus on four priority areas: whole-school implementation, School-to-Work 
transition, teacher assessment and certification, and development and use of scoring rubrics. 

This executive summary is organized by four clusters of research questions about use of the 
tailored CPA by ten schools in 1998-1999: (1) Tailoring the CPA, (2) Implementation, (3) 
Student performance and assessment, and (4) Whole-school change. The four clusters inform 
judgements about the tailored CPA’s scalability and potential for impact on students and schools. 
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Overview of the CPA 



In its initial development, the CPA was an interdisciplinary, performance-based assessment of 
California’s Career Preparation Standards (CPS). These standards set out the following skills 
critical to success after high school, whether in post-secondary education or in the workplace: 
Thinking and Problem-Solving, Communication, Technology Literacy, Personal; Interpersonal, 
Occupational Safety and Employment Literacy. 

The fixed-model CPA portfolio’s structure contains a set of cumulative tasks compiled by 
students during one or more semesters. It includes such tasks as a resume, a writing sample, a 
personal statement, and work samples. The tasks are flexible, so students can complete them in 
academic and/or career-technical education classes. Schools piloted the CPA in 1994, 1995, 
1996, 1998 and 1999. 

During the 1998-1999 school year, WestEd’s goal was to broaden the CPA’s scalability through 
the tailoring process. Ten high schools entered into partnerships with WestEd to develop and 
pilot a tailored portfolio design, giving them a key opportunity to alter the CPA to their own 
needs and standards, including designing portfolio guidelines for their students that define the 
portfolio’s standards and tasks, and contain the scoring rubric. Some 1,900 students from grades 
9-12 participated, with the number of juniors/seniors being slightly larger than 
freshmen/sophomores. Eighty-six teachers participated. 

In four schools, students completed substantial numbers of portfolios and full-blown scoring 
sessions. A fifth school took the intermediate step of creating benchmark portfolios to serve as 
exemplars in the classroom and in professional development activities. Three postponed 
in-depth CPA implementation until the 1999-2000 school year, but did not implement it then. 
Two schools did not implement the CPA in any meaningful way in 1998-1999 and did not 
participate in the 1999-2000 school year. Eight schools implemented the tailored CPA in 1999- 
2000. As a group, these schools were more successful than the previous cohort. Only 
preliminary data about the 1999-2000 group are available for this report. 

Methodology 

Data Sources and Instruments: 

• student portfolios from four schools in the 1998-1999 cohort (student portfolios from the 
1999-2000 cohort were scored, but serve as a limited data source for the report) 

• scorer evaluations 

• WestEd staff observations of scoring sessions 

• teacher surveys from eight schools (response rate of 68.6 percent) 

• interviews with teachers and principals/lead teachers 

• student survey from seven schools (response rate of 60.2 percent) 

• interim progress reports submitted to OERI 
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Approach: Data were aggregated by schools in which the portfolio completion rate was high 
enough to warrant scoring and those in which it was too low to score meaningfully. This was 
done because high completion rate schools can be termed “successful,” and much of what this 
report is about is identifying key success factors that influence scalability and impact. 
Furthermore, high completion rate (HCR) schools performed similarly to one another in terms of 
survey responses and other indicators. Correspondingly, low completion rate (LCR) schools 
performed similarly to one another. Four schools were HCR and six schools were LCR. Two of 
the six LCR schools failed to implement the CPA beyond very minimal levels and are excluded 
from analyses. 

Findings 

I. Tailoring the CPA 

Tailoring is a collaborative effort to customize the CPA, involving representatives of the school 
community, with WestEd staff acting as facilitators. Teachers, administrators, and community 
members work together to complete the following fundamental steps: 

• Analyze the original CPA portfolio standards (CPS) and the school’s own interdisciplinary 
standards or student outcomes in order to identify overlaps and differences. 

• Revise the CPS and the CPA scoring rubric to include local standards that are appropriate for 
measurement in a portfolio and to remove any unwanted CPS. 

Alter the existing CPA portfolio structure so that the tasks elicit substantial student work 
relating to the revised standards. 

Does the Tailoring Process Increase Teacher Buy-in? 

Interviewers asked teachers whether the tailoring process mattered to them and to other teachers. 
All interviewees responded positively to this line of questioning. The benefits teachers saw for 
themselves and others were: improved buy-in and ownership, relevance to the school and 
community, and relevance to the program. Data indicate that teachers, who did not participate in 
the tailoring process, and the accompanying professional development, did not have as much 
ownership of the CPA as did those who actually worked on tailoring the document. 

II. Implementation 

What Are the Key Conditions for Successful Implementation of the CPA? 

Prior to presenting data and analyses about this question, it is important to note that 
implementing portfolios school-wide at the high school level, and particularly in large, 
traditionally structured high schools, has proven to be quite difficult. Few high schools can 
generate the teacher buy-in to try, and even those find the curricular and logistical issues 
(communicating the portfolio’s purpose, explaining content and performance standards to 



O 

ERIC 



.L . 8 



students, coordinating portfolio tasks across disciplines, professional development, portfolio 
storage, setting up scoring sessions, etc.) daunting. 

Factors that contributed to the success of classroom implementation: Teacher surveys 
explored the question by asking respondents to identify what they saw as the five most important 
factors or resources that contributed to the success of the CPA in their classrooms during the 
1998-1999 school year. In LCR schools, the factors most often selected as contributing to 
“success” were: 

• experience with standards (55 percent) 

• school support of standards (52 percent) 

• significant numbers of students in common (38 percent) 

• student guidelines (36 percent) 

• professional development related to the CPA (36 percent) 

In HCR schools, the “success” factors most often selected were: 

• significant numbers of students in common (65 percent) 

• team teaching (59 percent) 

• integrated curriculum (58 percent) 

• block scheduling (41 percent) 

• career academies (41 percent) 

The data are characterized by their differences between the two groups. With the exception of 
“significant numbers of students in common with other teachers using the CPA,” the selections 
by teachers in the LCR schools were substantially different than those by teachers in the HCR 
schools. This can be explained in part by examining the structure of the four HCR schools. 

They share some common features: School-to-Work focus, integrated curriculum, and significant 
numbers of students in common. In essence, the factors selected by teachers in HCR schools are 
related to the structures of their schools and, in particular, structures that permit and require the 
cooperation of teachers across disciplines for a common cohort of students. It must also be noted 
that the enabling structures at the HCR schools are mostly present in schools that can be 
characterized as “reforming.” 

Schools’ ability to set up and implement portfolio scoring: Eighty-seven percent of the 
respondents expressed confidence that the school would be able to effectively conduct 
scoring/benchmarking next year without assistance/training from WestEd staff. However, 
WestEd staff noted that assembling scorers challenged most schools. More than 90 percent of 
respondents who completed scorer surveys felt the content of the training and the time devoted to 
training for scoring/benchmarking the portfolios was adequate. 
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Did Teachers and Students Value the CPA? 



Teacher and student perceptions of the benehts of the CPA: Teachers and students saw 
many benefits to the CPA. During interviews with teachers, respondents were asked to cite the 
primary purpose of the CPA. The most frequent response was that it benefited students. There 
was strong agreement among HCR and LCR teacher survey respondents that the skills upon 
which the CPA was based represented skills valued by both employers and post- secondary 
institutions (98 percent and 100 percent). Students also agreed (78 percent to 85 percent). 
Students and teachers in HCR and LCR schools were least optimistic about the actual completed 
portfolio’s value to post-secondary institutions (54 percent to 67 percent agreed). Survey data 
indicate that HCR students and teachers were consistently more positive than LCR students and 
teachers. 

Interviews with teachers and principals/lead teachers elicited comments about the CPA’s impact 
on students. In addition to the feeling that the CPA would help with access to jobs and college, 
teachers and principals/lead teachers believed there was an intrinsic benefit to students. They 
pointed to the pride students took in completing their entries, the sense of achievement they 
gained as they demonstrated mastery of the skills, and the personal growth that occurred as a 
result of the reflective component of the CPA. Some respondents also commented on the stress 
that the portfolio imposed on students, but they felt that this would be alleviated in the future by 
beginning implementation earlier in the school year. 

How Can the Implementation of the CPA be Improved? 

Teachers’ surveys explored the question by asking teachers to identify what they saw as the five 
most important factors or resources that would have made the CPA more successful. In other 
words, teachers identified the resources they didn^ have (or have enough of ) but thought would 
have been useful. 

The top five factors for LCR schools were: 

1. More teacher experience with the CPA (50 percent) 

2. Students with more experience with portfolios (43 percent) 

3. More school-wide support of the CPA (43 percent) 

4. More class time for the CPA (38 percent) 

5. More professional development related to the CPA (31 percent). 

« 

The top six factors for HCR schools were: 

1. More teacher experience with the CPA (59 percent) 

2. Students with more experience with portfolios (53 percent) 

3. More school-wide support of the CPA (29 percent) 

4. More school-wide support of the standards (29 percent) 

5. More experience with the standards (29 percent) 

6. Integrated curriculum (29 percent) 



Student surveys solicited opinions on how to improve the implementation of the CPA. In the 
LCR schools, “more examples” received the highest number of student responses (40 percent); in 
the HCR schools, the highest number of student responses (52 percent) was “more class time.” It 
is evident that in both LCR and HCR schools, a sizable number of students reported they did not 
get all the information, help, and support they needed: 35 percent in the LCR schools and 23 
percent in the HCR schools. 

Did Teachers Receive Appropriate Support to Develop and Implement a CPA 
Tailored to Their Schools’ Needs and Standards? 

Overall, more than 90 percent of the HCR teachers reported that the WestEd workshops prepared 
them adequately to implement the CPA in their classrooms. Nearly one-fourth of LCR teachers 
reported they were not well prepared to implement the CPA. Teachers were asked during 
interviews about ways to improve the professional development provided by WestEd. Half of 
the respondents cited issues that can be grouped under a category called “leadership.” This 
includes communicating expectations to teachers and students, communicating rewards and 
consequences, and conducting more follow-up. Half of the teachers interviewed had no 
suggestions for improving the professional development. 

III. Student Performance and Assessment 

The primary data sources are the results of the scoring of 287 portfolios at four schools in the 
spring of 1999 and questionnaires from scorers. 

The Scoring Process 

In an ideal scoring session, scorers would have a lengthy training followed by a series of pre- 
scored “calibration” portfolios that they would have to score accurately before being allowed to 
proceed to independent scoring. Then scorers would periodically stop scoring and recalibrate to 
ensure that their interpretation of the rubric had not “drifted.” The reality for the tailored CPA 
scoring sessions was different. The total length of time that schools arranged for scoring was 
only one day, often ending in mid-afternoon. Therefore, lengthy and appropriate training was not 
possible. In addition to limited training time, it was also not feasible to calibrate scorers. 

Teachers, parents, and other community members served as scorers. The scoring sessions began 
with a brief introduction to the portfolio and the scoring process. Once scorers were familiar 
with the scoring rubric and the anchor portfolios, they were given a set of three practice 
portfolios to score and discuss. In most cases training took two hours. In sum, these conditions 
naturally led to lower-than-desired reliability . Schools implementing tailored portfolios in the 
1999-2000 school year scored under better conditions and achieved higher reliability. 

Performance levels and rubric dimensions: Each school used three levels of performance to 
give scores on the dimensions of the rubric and an overall score, typically “Basic,” “Proficient,” 
and “Advanced.” Because the portfolio guidelines and scoring rubrics had been tailored to the 
assessment priorities of each school, there was some variation in the dimensions of student 



performance being assessed. However, all schools addressed dimensions related to career 
preparation, technology, analysis, and communication. 

How reliable were the tailored portfolio as assessments? 

In our analyses, inter-rater reliability was calculated in two ways: first, in terms of exact 
agreement within a scoring level, and second, allowing agreement across main performance 
levels by use of the plus and minus scores. Under the first method, scores were counted as 
agreeing if they were in the same performance level. For example, scores of Proficient minus 
and Proficient plus would count as agreement, but Basic plus and Proficient minus would not , 
even though Basic plus and Proficient minus are “closer” together than Proficient minus and 
Proficient plus. 

Under the second method, scores with exact agreements and scores of plus on one level and 
minus on the level above were regarded as agreeing. For example, Basic plus and Proficient 
minus were counted as agreeing, but Basic plus and Proficient were not. Figure A shows the 
inter-rater reliabilities for the overall portfolio (versus dimensional) scores under each method, 
and the range of inter-rater reliabilities calculated for individual scorers. The reason for using 
two methods was to see if the use of plus and minus scores improved the agreement of scorers 
when scorers were divided across performance levels, but close (e.g., Basic plus and Proficient 
minus). 

Figure A: Inter-rater reliability of overall scores 





Inter-rater Reliability 
(method one, agreement 
within a level) 


Inter-rater Reliability 
(method two, agreement 
within a plus or minus) 


School 


Overall 

% 


Min/Max 
IRR per 
scorer 
% 


Overall 

% 


Min/Max 
IRR per 
scorer 
% 


Drake 


57.89 


33.3 / 83.3 


66.67 


33.3/ 83.3 


New Technology 


68.38 


38.9/84.2 


73.5 


39.5/85.8 


Peoria 


69.51 


25.0/ 100.0 


75.61 


25.0/ 100.0 


San Leandro 


51.47 


30.8/ 100.0 


52.94 


30.8/ 100.0 



Inter-rater reliabilities were not as high as desirable, but were understandable given the difficult 
scoring conditions. Using method two, where score agreements are allowed across the level 
boundaries, produced slightly higher reliabilities. Using method one inter-rater reliabilities 
ranged from 57.89 percent to 69.51 percent. Using method two they ranged from 52.94 percent 
to 75.61 percent. Reliabilities for individual scorers varied greatly, as opposed to scorers at a 
given school taken as a group. 





As a whole, reliability data suggest that improving scoring conditions is necessary for the 
portfolio to be used in high-stakes environments. However, it is likely that via increased training 
time and other steps, reliability could increase so that high-stakes use of the portfolio is 
appropriate, as part of a multiple-measure system. Critically, current levels of reliability may not 
significantly impact scalability — no 1998-1999 schools indicated that lower than desired 
reliability would decrease implementation. In fact. HCR schools generally expanded 
implementation . 

Increasing reliability 

While no definitiye data exist about why reliability was lower than desired, or why it yaried 
across schools, WestEd staff hypothesize that adhering to the following scoring procedures 
would substantially increase reliability: 

• Longer training. 

• Target training to dimensions of the rubric that had the lowest inter-rater reliabilities. 

• Examine the wording of dimensions of the rubric that had the lowest inter-rater reliabilities 
and rewrite where appropriate. 

In 1999-2000 scoring sessions, implementing those procedures resulted in increased reliability (a 
range of 64 percent to 79 percent in 1999-2000 yersus a range of 58 percent to 71 percent in 
1998-1999). 

How Well Did Students Perform? 

In the simplest assessment of student performance. 68 percent of students who completed 
portfolios “passed.” or receiyed a Proficient or Adyanced score, and 32 percent “failed” or 
receiyed a Basic . In three of the schools, the majority of scores fell within the Proficient 
category. When considering the strength of these scores, the reader should know that only 
“complete” portfolios (yersus incomplete or portfolios where not all entries were done) were 
scored. This is important as quality correlates with completeness — students with the skills and 
attitudes to complete a portfolio typically submit higher quality entries than those that do not 
complete portfolios. 

The distribution of scores across the four schools was fairly similar, with the exception of Peoria. 
(Peoria had 58.9 percent Basic, as opposed to an ayerage of 23.2 percent for the other three 
schools.) Excluding Peoria, the range of Basic was 17.4 percent to 30.2 percent; Proficient 51.8 
percent to 58.1 percent; and Adyanced 11.8 percent to 26.4 percent. In comparison to the 1996- 
97 cohort, the oyerall percentage of Proficient scores was the same at 50 percent. Basic and 
Adyanced percentages were also similar. 

While the number of schools, students, and portfolios participating in 1996-97 and 1998-1999 is 
not large, data suggest that schools possessing the factors for successful implementation 
identified in 1996-97 and confirmed in 1998-1999 might expect student performance to fall 
within the ranges seen in 1996-97 and 1998-1999 . 
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IV. Whole-school change 

What is the Impact of the Tailored CPA on Teacher and School Practices? 

More than 70 percent of teachers in both LCR and HCR schools reported they engaged in the 
following teacher practices: (1) they made substantial use of the standards to structure their 
curriculum, (2) they made substantial changes in their teaching methods as a result of using the 
standards, and (3) they made substantial changes in their teaching methods as a result of using 
the CPA. Teachers in HCR schools attributed more changes in their teaching practices to the 
CPA and students’ efforts for the CPA than did teachers in LCR schools. 

The student survey elicited comparative responses about teacher practices. When asked if 
teachers who use the CPA teach differently than non-CPA teachers, 46 percent of LCR students 
agreed or strongly agreed. At HCR schools, 68 percent agreed or strongly agreed and the percent 
of students that strongly agreed was triple that of the LCR schools. When asked if CPA teachers 
gave different assignments than non-CPA teachers, 52 percent of LCR students agreed/strongly 
agreed and 64 percent of HCR students agreed/strongly agreed. 

What is the Impact of the CPA on Teacher Understanding of Standards- 
Based Teaching, Assessment, and Portfolios? 

Data pertaining to the question come from interviews with teachers and principals/lead teachers. 
Nine of the 16 teachers interviewed reported the CPA had no impact on their understanding of 
standards-based teaching, assessment, and portfolios because they came into the program with a 
high level of understanding in each of the areas. The remaining seven teachers reported they had 
experienced personal growth and understanding and specifically mentioned a deeper 
understanding of assessment and the relationship between standards and performance. All of the 
principals/lead teachers interviewed felt the teachers in their schools had a greater understanding 
of standards-based teaching, assessment, or portfolios. 

What is the Impact of the CPA on Whole-school change. Particularly in 
Schools with Traditional Structures? 

Teacher surveys indicate there is general agreement in LCR and HCR schools that the CPA is 
valuable for implementing standards-based curriculum and assessment school-wide. Teachers in 
HCR schools, however, appear to feel much more strongly about the CPA as a valuable support 
for implementing standards-based curriculum and assessment school-wide (HCR 47 percent 
strongly agree, LCR 15 percent strongly agree). 

In summary, there was general agreement in LCR and HCR schools that the CPA is a valuable 
tool for implementing standards-based curriculum and assessment school-wide. However, in 
terms of actually causing, versus supporting, substantial change in structures, procedures, or 
curriculum, teachers were less conclusive. It appears that in LCR schools, teachers that bought in 
to the CPA did make changes. LCR teachers that did not buy in, and they appeared to be the 
majority at LCR schools, did not make changes. At HCR schools, deep structural and classroom 
changes were less likely as they already had enabling structures in place. 






CONCLUSION 



The 1998-1999 CPA effort generated data about the potential of the tailored version of the CPA 
to be used as a scalable, reliable assessment that supports systemic reform. Like the whole of this 
report, the conclusion is organized around the four research areas of tailoring, implementation, 
student performance and assessment, and whole-school change. 

Tailoring 

1. The tailoring process enabled schools to efficiently customize the CPA to their own needs, 
thus increasing ownership or “buy-in.” 

2. The tailoring process appears to have ongoing substantive, in addition to buy-in, value. 

3. The tailoring process did not place undue burden on schools. 

4. Tailoring is integral to long-term scalability due to its impact on buy-in and implementation. 
Implementation 

1. The tailored CPA can be implemented successfully in schools possessing certain structural 
characteristics generally associated with schools that are engaged in whole-school reform. 
HCR schools possessed a preponderance of these conditions while LCR schools possessed 
few. Four schools started the CPA in the 1999-2000 school year and were chosen with more 
emphasis on the structural characteristics listed below than the schools chosen for 1998-1999. 
All four could be characterized as HCR and will continue to implement in the 2000-2001 
school year. The key characteristics are: 

• recognition of the benefit of standards-based teaching and assessment for students 

• strong teacher buy-in for the CPA as the right vehicle to implement standards-based 
teaching and assessment 

• strong school leadership 

• commitment to reform 

• whole-school focus by teachers (as opposed to strong focus on individual departments) 

• integrated curriculum 

• interdisciplinary/team teaching 

• block scheduling 

• a shared cohort of students 
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2. In CPA pilot tests over four years these conditions were most prevalent in small schools, or 
smaller units within schools (e.g., academies, career paths). This suggests that the CPA is 
most sustainable and scalable in smaller learning environments. 

3. Like other substantial assessment and curriculum tools that require schools to be well down 
the reform path, the CPA is not likely to be scalable across a majority of high schools given 
the current reform status of most high schools. 

4. As in 1996-1997, survey and interview data from teachers and students were positive about 
the value of the CPA. 

5. Scoring large numbers of portfolios challenges schools’ ability to provide scorers and could 
impact scalability. 

Student Performance and Assessment 

1. Reliable scoring of the tailored CPA requires a commitment to conducting scoring with 
substantial training. 

2. Student performance on the tailored CPA was acceptable to implementing teachers. 
Whole-school change 

1. A strong majority of 1998-1999 project teachers believe the tailored CPA is a valuable 
teaching tool that promotes change in classroom practice and can support school-wide 
reform. 

■ 2. The tailored CPA allowed and supported “ready” schools to implement their visions related 

to having a school-wide student performance focus and to standards-based teaching and 
learning. 

The capacity of any school not possessing a preponderance of the CPA’s conditions for success 
to implement any type of standards-based reform on a school-wide basis is highly questionable. 
Given that, the fact that the CPA did not fare well in LCR schools should not be surprising or 
disconcerting. Moreover, that the CPA is flourishing in schools with faculties acutely attuned to 
innovation, and in schools possessing the conditions for success is strong testimony to its value. 
In our national context where precious few high schools are implementing standards-based 
reform in more than a few departments, the CPA is an effective tool to help scale standards-based 
teaching and learning across whole schools and across ready subsets of schools. 
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Introduction 



Systemic reform aimed at improving student performance requires schools and school systems to 
change a wide variety of practices. This characteristic compels educators to use practices that 
facilitate change across multiple dimensions of schooling; for example, teaching and learning, 
curriculum, assessment, grading, and governance. In 1994, WestEd embarked on an effort to 
develop a scalable, reliable, performance-based assessment that supported systemic reform 
(particularly reform informed by School-to-Career principles) and improved student 
achievement. WestEd created a standards-based portfolio of well-defined student tasks called the 
Career Preparation Assessment portfolio, or CPA, variations of which have been implemented 
and tested in schools since the 1994-1995 school year. The original CPA model focused on a 
fixed set of standards that aligned with other important career-related skills and standards 
frameworks such as the SCANS report. 

Over the first three years of the CPA’s implementation, WestEd saw that the fixed-model 
approach hindered the portfolio’s ability to be accepted by entire faculties and to be integrated 
into local reform efforts. High school faculties wanted the ability to alter the portfolio’s tasks 
and standards. In response, in the 1998-1999 school year, WestEd developed a method for 
tailoring the CPA’s standards and components to meet local needs. WestEd made this move to a 
tailored CPA in an attempt to increase: 

• the pool of schools interested in using the CPA as a high-stakes assessment (tailoring 
increased teacher buy-in and broadened the CPA’s applicability to non-School-to-Work 
settings), 

• educator desire to use performance on the CPA as meaningful data informing systemic 
change, 

• employer desire to use the CPA in hiring decisions, and 

• admissions officers’ desire to use it in admissions decisions. 

The tailoring process, which will be described in detail later in this report, required WestEd to 
increase the amount of resources it provided to implementing schools. It involved teachers, 
administrators, and others in the school community, thereby strengthening commitment to the 
CPA. WestEd viewed this investment as likely to increase, rather than decrease, scalability — 
without thorough faculty buy-in, any major reform is doomed. 

This report is organized by four clusters of research questions about use of the tailored CPA by 
ten schools in 1998-1999: (1) Tailoring the CPA, (2) Implementation, (3) Student performance 
and assessment, and (4) Whole-school change. The four clusters all inform judgements about 
the tailored CPA’s scalability and potential for impact on students and schools. 
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Implementation: This is a critical topic because practitioners considering interdisciplinary, 
standards-based curriculum and assessment must know more about what it takes to achieve 
success. Also, to make judgments about the CPA’s scalability, it is important to identify key 
conditions for successful implementation of the CPA as well as teacher and student views about 
the utility of the CPA. 

Student performance and assessment: The quality of the tailored CPA as an assessment tool 
needs to be understood in order to judge its usefulness as a data source to inform change at the 
classroom, school, and district levels. Furthermore, analyzing student performance on the CPA 
will tell much about its utility and implementability as an assessment and a curricular tool. 

Tailoring: It is important for practitioners to understand which adaptation and implementation 
approaches lead to maximum success. Data on the implementation of the tailoring process will 
help to identify successful strategies for improving teacher and administrator commitment 
through customizing the CPA to meet local needs of schools and districts. 

Whole-school change: The actual impact of the CPA on whole-school change needs to be 
understood so that the CPA’s overall value in the reform context can be described accurately and 
so that practitioners can make judgments about the need for complementary instruments or 
procedures. 

Study of, and support for, the CPA aligns closely with WestEd’s approach to its national 
specialty area of assessment — improving student performance and bolstering whole-school 
change through focus on four priority areas: whole-school implementation, School-to-Work 
transition, teacher assessment and certification, and development and use of scoring rubrics. 
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Overview of the Career Preparation 
Assessment 

In its initial development, the CPA was an interdisciplinary, performance-based assessment of 
California’s Career Preparation Standards (CPS). These standards set out the following skills 
critical to success after high school, whether in post-secondary education or in the workplace: 
Thinking and Problem-Solving, Communication, Technology Literacy, Personal, Interpersonal, 
Occupational Safety, and Employment Literacy (Appendix A). These standards align with most 
high schools’ own expected student outcomes and with important STW standards frameworks, 
including those in the SCANS report. The CPA provides a bridge across the theoretically 
interrelated reform vehicles of STW, standards-driven curriculum and assessment, and integrated 
curriculum. 

The model CPA portfolio’s structure, developed by WestEd in collaboration with teachers and 
employers, contains a set of cumulative tasks compiled by students during one or more 
semesters. The completed portfolio includes such tasks as a resume, a writing sample, a personal 
statement, and work samples. The tasks are flexible, so students can complete them in academic 
and/or career-technical education classes. Because students can demonstrate one or more 
standards (e.g., problem-solving or communication skills) in almost any class, teachers from all 
disciplines can support the CPA as a useful tool for standards-based instruction and assessment, 
as well as a vehicle for supporting whole-school change. The entries are presented in Table 1 on 
the following page. 

The purposes of the CPA are: (1) to foster and assess student achievement of skills critical to 
success after high school, (2) to bolster standards-driven reform efforts that focus on preparing 
students for the post-high school world, (3) to provide a foundation for teacher collaboration and 
interdisciplinary curriculum, and (4) to provide students with a useful tool in applying for either 
entry-level employment or admission to post-secondary education — a completed CPA portfolio. 



THIS SPACE INTENTIONALLY BLANK 





3 



Table 1: Description of Cl*A entries 



Entry 


Description 


Personal statement 


Students outline their career goals and evaluate their skills. 


Resume 


Students prepare a one-page resume describing their experiences and 
skills. 


Application 


Students obtain and complete an application for employment or continued 
educational training. 


Letter of 
recommendation 


Students obtain a letter of recommendation from someone who 
knows them well, such as a supervisor, community leader, or 
teacher. 


Work samples 


Students select pieces of work demonstrating their mastery of the CPS. 
Work samples can range from a science experiment to organizing a 
community event to a statistical analysis of a school-wide survey. (One of 
the two work samples must address Technology Literacy.) 


Writing sample 


Students select any document demonstrating their ability to communicate 
in writing and to reach a conclusion based on supporting information and 
evidence. Students are evaluated on their writing ability and analytical 
reasoning. Writing samples can range from a comparative analysis of short 
stories to a business proposal. 


Interpersonal skills 
evaluation 


An evaluation of students’ interpersonal skills (team work, leadership, 
etc.), the ISE is completed by a supervisor or teacher after a work 
experience, a team project, or a class. 


Optional entries 


Students may create an additional section in their portfolios and include 
any of the following: 1) attendance records, 2) transcript with GPA, 3) 
extra-curricular activities, certificates, and awards, or 4) cover letter. The 
CPA rubric does not address optional entries. 



20 



The power of the tailored CPA’s approach is fourfold: 

1. The tailoring process and use of the portfolio improves teachers’ expertise in and 
commitment to teaching and assessing standards. 

2. The portfolio’s interdisciplinary nature creates the basis for comparison of student work and 
tasks across a school’s curriculum, thereby providing a vehicle for all teachers to look at 
students in terms of school-wide, versus a single discipline’s, student outcomes. 
Concomitantly, the portfolio’s interdisciplinary nature encourages teachers to collaborate 
across disciplines so that students attain those school-wide outcomes. Moreover, the CPA’s 
interdisciplinary nature helps build a learning community among teachers that supports 
school-wide reform. Implementing the CPA promotes a whole-school dialog about student 
achievement and, in particular, about defining what students should know and be able to do. 

3. Unlike many school-wide assessments (e.g., standardized tests), teachers play a major role in 
selecting and constructing this assessment. The teacher role increases the likelihood that 
they will use performance data from across the curriculum to inform their teaching. 

4. Because the value of the portfolio as a whole is greater than the sum of its parts and because 
it focuses on skills needed for life after high school, students become more motivated to 
achieve the portfolio’s standards. 



History 

1994-1995 

Development of the CPA began in the winter of 1994 when California’s legislature passed AB 
198, which mandated schools to report on students’ readiness for the world of work. The 
California Department of Education (CDE) then contracted with WestEd to develop an 
assessment which: (1) supported high schools’ compliance with the new law, (2) aligned with the 
state’s new high school reform vision (Second to None), and (3) improved student attainment of 
the Career Preparation Standards, California’s equivalent of the SCANS skills and competencies. 
The new assessment was to build on and align with WestEd’ s prior pioneering work with the 
CDE in the field of performance-based assessment of occupational cluster standards, the Career- 
Technical Assessment Program (C-TAP). 

During the winter of 1994, WestEd staff, in collaboration with educators and employers, 
designed the entries that comprise the model CPA portfolio, and they compiled the student 
guidelines that assist students and teachers in completing the entries. WestEd recruited eight 
schools to pilot test portfolio entries (not the complete portfolio) during the spring of 1 995. 
Nevertheless, students from two of the pilot sites submitted complete portfolios. WestEd staff, 
pilot test teachers, and employers with experience in entry-level hiring evaluated the more than 
2,000 completed portfolio entries, creating a number of portfolios as examples of the “Proficient 
“and “Advanced” levels for use in teacher training and with students. 
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In addition to analyzing entries, the following key activities took place; 

• WestEd developed and analyzed student and teacher surveys regarding the utility and 
impact of the CPA. Major findings of the 1994-1995 surveys were that students and 
teachers found the CPA easy to use, that the CPA was a powerful teaching and 
learning tool, and that many students saw a clear tie between success with the CPA 
and preparedness for the world of work. 

• The student guidelines for implementing the CPA were revised using data and 
feedback from pilot test teachers. 

• A scoring rubric, aligned tightly with the student work received in the pilot test, was 
drafted and refined after review by practitioners and employers. 

• Dissemination work began aggressively, with presentations at major assessment and 
systemic reform conferences and a mailing to some 200 targeted high schools. 

From the work in 1994-1995, WestEd learned that the CPA had strong potential as an 
implementable, comprehensive assessment tool that organically promotes curriculum integration 
and teacher collaboration. Also, the work generated initial data about some preconditions for 
successful implementation. Schools with supportive stmctures (integrated curriculum, team 
teaching, common planning time) and prior experience with performance-based assessment 
submitted more and higher quality student work. Overall, the 1994-1995 CPA effort helped to 
prepare WestEd to launch a pilot test of the model where students would be required to create 
“complete” portfolios where all entries were done. WestEd knew a good deal about what types 
of schools were likely to succeed at CPA implementation and therefore about what schools to 
select for that pilot. 

1995 - 1996 

In the 1995-1996 school year, state support for the CPA was withdrawn following a major 
restructuring of the CDE. However, the two schools that submitted complete portfolios in 
1994-1995 and one other school from the first year continued to pilot test the CPA with support 
from WestEd. Some 150 completed portfolios were submitted by the three schools. With 
support from the U. S. Department of Education’s Office of Educational Research and 
Improvement (OERI), the portfolios were evaluated, exemplars selected, and modifications to the 
guidelines, scoring rubric, and professional development were made based on teacher and student 
surveys and other feedback. Also, teachers from some of these pilot schools provided 
substantive input into the design of the 1996-1997 pilot test, adding new information about key 
conditions for successful completion of the CPA portfolio. The 1995-1996 effort, while limited, 
further readied WestEd to hold a full pilot test where relatively large numbers of teachers and 
students had a full year to complete portfolios. 

1996 - 1997 

The 1996-1997 school year was pivotal for the CPA. Limited pilot testing in 1994 and 1995 had 
been successful, and informal data gathering suggested that high schools across California were 
at a beginning readiness level to try a performance-based assessment of generic work readiness 
skills. WestEd decided to launch a significant test of the CPA across several factors: usability, 
reliability as an assessment, and impact of use on whole-school change. 




6 



22 



The 1996-1997 pilot testing of the CPA was designed to build on development work completed 
during the prior school years. To start, WestEd identified 300 high schools likely to have interest 
in, and capacity for, implementing the CPA and invited them to apply for participation in the 
1996-1997 pilot test. Twenty-eight schools submitted four-page applications. WestEd selected 
seven that appeared able to implement well and represented diversity of geography and ethnicity. 
Key factors in selection were: (1) number of teachers and students who were to participate; (2) 
teacher experience with performance-based assessment, working in teams, and with integrated 
curriculum; (3) teacher commitment to implementation; and (4) number of shared students. The 
quid pro quo for participation in this larger pilot test was that WestEd would provide materials 
and professional development, while participating teachers and students would submit their 
portfolios and observations about implementing the CPA to WestEd for analysis. 

The selected schools ranged widely in terms of geography, experience with performance-based 
assessment, and structure. All but one of the schools were ethnically diverse. All schools 
accepted into the pilot, were judged to have the capacity to have their students produce substantial 
numbers of complete portfolios — no schools were accepted that appeared to lack the 
fundamentals necessary for completing portfolios. Six schools (one of the original seven schools 
dropped out prior to the beginning of the 1996-1997 school year), 36 teachers, and over 600 
students participated. The six schools received professional development about the CPA from 
WestEd. 

Conducting a substantial field test in 1996-1997 gave WestEd its first fairly complete view into 
CPA implementation across a range of schools and across schools that were implementing under 
good circumstances. The data gathered from teachers, students, and the portfolios themselves 
allowed WestEd to make informed decisions about revising the CPA, requirements for future 
participation from schools, and teacher training. 



1997-1998 

During the 1997-1998 school year, WestEd used the 1996-1997 field test findings to revise and 
refine the implementation process. In particular, WestEd reviewed data indicating that the fixed- 
model approach hindered the portfolio’s ability to be accepted by entire faculties and to be 
integrated into local reform efforts. This decreased the CPA’s potential impact and scalability. 
High school faculties wanted the ability to alter the portfolio’s tasks and standards, even if 
alterations were largely symbolic in nature. WestEd created a process for tailoring the standards 
and components that make up the CPA to accommodate schools’ needs and standards. Ten high 
schools were recruited in the spring of 1998 to participate in a two-year trial of the tailored 
version of the CPA. Participation requirements for this trial were substantially higher than in 
prior years. Participating schools committed to: 

1 . work as part of a team with WestEd; 

2. have a deep commitment to measuring interdisciplinary standards that relate to the following 
skills: Thinking and Problem-Solving, Communication, Technology Literacy, Personal, 
Interpersonal, and Employment Literacy; 

3. expend significant resources (4-6 days of release time — for professional development for 
teachers using the portfolio, to collaborate with WestEd in portfolio design, and to score 
portfolios) toward making the portfolio a success; 
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4. have thorough “buy-in” for the portfolio from teachers who will be implementing it; 

5. recognize that successful implementation of their portfolio requires a major, sustained effort 
on the part of teachers and students; 

6. implement the portfolio for two years (for example, all 1998-2000 juniors in their junior and 
senior years); and 

7. in the second year of the partnership, place significant stakes on student performance with 
their portfolio (possibilities include: 50 percent of the grade in multiple classes, graduation 
requirement, prominent inclusion of performance on transcript). 

By the end of 1997-1998, WestEd had a diverse set of schools on board to pilot test the tailored 
model. 

1998-1999 

During the 1998-1999 school year, WestEd’s goal was to broaden the CPA’s scalability through 
the tailoring process. Ten high schools entered into partnerships with WestEd to develop and 
pilot a tailored portfolio design, giving them a key opportunity to alter the CPA to their own 
needs and standards, including designing portfolio guidelines for their students that define the 
portfolio’s standards and tasks, and contain the scoring rubric. The guidelines are designed to be 
a stand-alone document that provides students with all the information they need to complete the 
portfolio. Teachers created their schools’ guidelines by tailoring the fixed-model CPA guidelines 
to their school’s scoring rubric. All ten schools chose to adopt most of the Career Preparation 
Standards. WestEd’s ongoing professional development activities with partner schools focused 
on implementation support, preparation for scoring, and the logistics of portfolio storage. 

At all ten schools implementation was interdisciplinary, generally by teachers from academic and 
career-technical disciplines. For example, the personal statement would be completed in 
English, one work sample in Math and the other work sample in Agriculture. Implementation 
generally began late in the first semester or early in the second, and continued throughout the 
year. Some 1,900 students from grades 9-12 participated, with the number of juniors/seniors 
being slightly larger than freshmen/sophomores. Eighty-six teachers participated. Of the ten 
schools, three implemented in traditional structures, three in academies, two in academies within 
a traditional structure, and two in alternative/continuation schools. 

Teachers primarily implemented the portfolio as a teaching and learning tool in the 1998-1999 
school year, gaining confidence with both the materials and the portfolio development process. 
They also prepared to add higher stakes to the mix during the second year (1999-2000) when 
meeting the portfolio’s performance standards was to become a graduation requirement or a 
significant part of students’ grades. 

In four schools, students completed substantial numbers of portfolios and full-blown scoring 
sessions were held involving teachers, school and district administrators, school board members, 
employers, and parents. A fifth school took the intermediate step of creating benchmark 
portfolios to serve as exemplars in the classroom and in professional development activities. 
Three postponed in-depth CPA implementation until the 1999-2000 school year, but did not 
implement then. Two schools did not implement the CPA in any meaningful way in 1998-1999 
and did not participate in the 1999-2000 school year. 
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Completed portfolios typically were scored on site at the end of the school year by a team, 
usually consisting of teachers and other district personnel. In some cases community members 
were invited to participate. The team met prior to the scoring for training in the scoring rubric. 
(See Appendix B for the model CPA rubric; see Appendix C for examples of tailored rubrics; see 
Appendix D for tailored standards.) 

Using tailored scoring rubrics, completed portfolios received an overall rating on a 3-point scale, 
typically labeled Basic, Proficient, or Advanced. The lowest, or “Basic,” rating serves as an 
indicator that the student has not mastered the specified standards, is not hirable for an entry- 
level job in a non-technical field, and that the portfolio is not ready to show to an entry-level 
employer or college admissions officer. The “Proficient” rating is an indicator that the student 
has mastered the specified standards, is hirable for an entry-level job in a non-technical field, and 
that the portfolio is ready to show to an entry-level employer or college admissions officer. The 
highest, or “Advanced,” rating is an indicator that the student has strong mastery of the specified 
standards, is hirable for an entry-level job in a non-technical field, and that the portfolio may 
exceed the expectations of an entry-level employer or college admissions officer. 

1999-2000 

For the 1999-2000 school year, WestEd created partnerships with four new high schools to 
develop and implement tailored portfolios. It maintained partnerships with the four 1998-1999 
schools that produced enough portfolios to hold scoring sessions. Thus, a total of eight sites 
participated in the 1999-2000 school year. The new sites were selected with increased emphasis 
on structural characteristics identified as key to successful implementation. In all eight schools, 
students completed substantial numbers of portfolios and full-blown scoring sessions were held 
involving teachers, school and district administrators, school board members, employers, and 
parents. WestEd provided the same support for 1999-2000 schools as it did for 1998-1999 
schools. 
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Methodology 

Throughout the 1998-1999 school year and into the following summer, WestEd gathered data to 
analyze the CPA in terms of the four broad research categories delineated earlier: (1) Tailoring 
the CPA, (2) Implementation, (3) Student performance and assessment, and (4) Whole-school 
change. 

Data Sources and Instruments 

Student Portfolios: Students from four 1998-1999 schools (New Technology — Napa, CA; 
Peoria — Peoria, AZ; San Leandro — San Leandro, CA; and Drake, San Anselmo, CA) completed 
substantial numbers of CPA portfolios. Portfolio scoring took place at the four schools in late 
May/early June 1999. WestEd staff provided training and supervised all the scoring sessions, 
which included teachers who implemented the CPA in their clcissrooms. Principals and other site 
administrators participated in some of the scoring sessions, as did district administrators and 
school board members. Finally, parents and employers from the community participated in 
scoring at some schools. In the six remaining schools, there was an insufficient sample of 
completed portfolios to conduct scoring. (Portfolios from the eight 1999-2000 schools were also 
scored in June of 2000. Only preliminary data are available about these portfolios.) 

Scored portfolios were given a score on each of the dimensions defined by the school’s scoring 
rubric (see Appendix B for examples of dimensions on the model CPA rubric: Career 
Preparation, Analysis, Technology and Communication). Student performance on these 
dimensions, or sub-categories within the rubric, informed the overall rating of a portfolio. The 
four schools all had the following dimensions in common: Career Preparation, Analysis, 
Technology, and Communication. As part of the tailoring process, some schools gave these 
dimensions different names, and schools added to or modified these dimensions. However, 
substantively, they were quite similar. In any case, the scoring rubrics contained descriptors of 
student performance related to one or more of the seven Career Preparation Standards at the 
Basic, Proficient, and Advanced levels. For example, a descriptor of written communication at 
the Advanced level is “Writing is clear and well organized throughout the portfolio.” The 
analogous descriptor at the Basic level is “Ideas are presented in a disorganized way.” 

In three schools, each completed portfolio was scored by at least two different scorers. In one 
school, a method of scoring was used that inadvertently resulted in some portfolios being scored 
by only one scorer while other portfolios were scored by three scorers. WestEd’s approach to 
scoring portfolios is more extensive than the scoring procedures generally used with portfolios. 

In most portfolio scoring systems, only one scorer scores each portfolio. A second scorer then 
scores a sample of the first scorer’s portfolios to be sure that the first scorer applied the scoring 
rubric correctly. Scoring many portfolios multiple times generated rich data, which WestEd used 
to explore issues related to the reliability with which CPA portfolios could be scored, and how 
dimension scores related to the overall scores given to portfolios. 
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The following number of portfolios were scored in 1998-1999 (includes anchor and training 
papers): 



New Tech 


81 


Peoria 


91 


San Leandro 


78 


Drake 


64 


Total 


314 



Scorer Evaluations: After the portfolios had been scored, each scorer responded to a one-page 
questionnaire asking whether the training conducted by WestEd prepared them adequately for 
scoring and benchmarking student portfolios, whether the training and scoring experience would 
carry over into future scoring of the CPA, whether the rubric should have three or four score 
points, and whether participating in scoring was good preparation for helping students produce 
Proficient portfolios. Scorers’ surveys were completed by 65 respondents. 

WestEd Observation of Scoring Session: WestEd staff took note of key developments during 
the scoring session. 

Teacher Survey: Near the end of the 1998-1999 school year, teachers in eight schools 
completed surveys about their experiences with the tailored CPA. The survey focused'on the 
following areas: 

• Resources which would have made CPA implementation more successful 

• The extent to which the CPA was implemented in the teacher’s classrooms 

• Overall impressions of the CPA 

• The degree to which using the CPA affected teaching methods or curriculum 

• The most important factors contributing to the successful implementation of the CPA 

• The impact of the tailoring process on teacher commitment (buy-in) 

• Implementation, alignment with school standards 

• Professional development and training for the CPA 

• Other education reform efforts occurring at the school, excluding the CPA 

• Work with standards, integrated curriculum, block scheduling, team teaching, and 
career academies 

Teachers responded to most of the items on a Likert-type scale which asked how strongly they 
agreed or disagreed with a statement. Teachers at the eight schools completed 59 surveys for a 
response rate of 68.6 percent. At two schools, teachers did not complete surveys. 

Interviews with Teachers and Principals/Lead Teachers: Telephone interviews were 
conducted in late May/early June 1999 with the principal or lead teacher from each of the ten 
schools. In addition, telephone interviews were conducted with 16 teachers from five schools. 

The purpose of the interviews was to obtain more detailed information about the issues examined 
in the teacher survey. 
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Student Survey: Students from seven schools completed a survey near the end of the 1998-1999 
school year that gathered information about the following areas: 

• their perceptions of the CPA, including their perception of the value of the CPA to 
employers and colleges. 

• how the CPA was implemented in their classrooms. Issues included whether they 
understood the standards and the CPA, the level of assistance they received from their 
teachers, and how classrooms where the CPA was used differed from other classrooms. 

Most of the items were on a four-point Likert-type scale which asked how strongly students 
agreed or disagreed with a particular statement. Some items also asked students to explain their 
responses. A total of 1 ,144 students responded to the survey for a response rate of 60.2 percent. 

In addition to the data sources described above, interim progress reports submitted to the U. S. 
Office of Educational Research and Improvement were a source of descriptive and anecdotal data 
about the implementation of the project. 

A copy of each data collection instrument may be found in Appendix E. 

Approach: Data were aggregated by schools in which the portfolio completion rate was high 
enough to warrant scoring and those in which it was too low to score meaningfully. This was 
done because high completion rate schools can be termed “successful,” and much of what this 
report is about is identifying key success factors that influence scalability and impact. 
Furthermore, high completion rate schools performed similarly to one another in terms of survey 
responses and other indicators. Correspondingly, low completion rate schools performed 
similarly to one another. Low completion rate schools will be referred to as LCR; high 
completion rate schools will be referred to as HCR. Four schools were HCR and six schools 
were LCR. Two of the six LCR schools failed to implement the CPA beyond very minimal 
levels and are excluded from this report’s analyses — these two schools did not implement enough 
to have informed opinions about use of the portfolio. 

This HCR/CCR approach provides two benefits. First, analyses resulting from the approach 
informing much of the report’s key conclusions about scalability — that the tailored CPA is 
scalable in schools that resemble HCR schools structurally. Second, it allows the reader through 
two sets of practitioner’s lenses, HCR and LCR. 
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Findings 

Findings are organized according to the research categories listed in the Introduction: 

(1) Tailoring the CPA, (2) Implementation, (3) Student performance and assessment, and (4) 
Whole-school change. 

I. Tailoring the CPA 

Tailoring is a collaborative effort to customize the CPA, involving representatives of the school 
community, with WestEd staff acting as facilitators. Teachers, administrators, and community 
members work together to complete the following fundamental steps: 

• Analyze the original CPA portfolio standards (CPS) and the school’s own 
interdisciplinary standards or student outcomes in order to identify overlaps and 
differences. 

• Revise the CPS and the CPA scoring rubric to include local standards that are 
appropriate for measurement in a portfolio and to remove any unwanted CPS. 

• Alter the existing CPA portfolio structure so that the tasks elicit substantial student 
work relating to the revised standards. 

In the tailoring process, WestEd staff conducted two full days of intensive on-site work at each 
school with teachers, school and district administrators, and representatives of the school 
community. They explored, in detail and in practical terms, how they could cooperate across 
subject areas to make the CPA work to help their students meet carefully designed performance 
standards. Following the two-day session in each school, WestEd staff revised the CPA 
guidelines to meet each school’s needs and standards. (See Appendix F for an example of 
tailored CPA guidelines.) Throughout this tailoring process, teachers continually redefined what 
they wanted students to know and be able to do, which standards should be taught in which 
classes, and which school structures, such as scheduling or student grouping, needed revision in 
order to support implementation of a school-wide curricular reform effort. 

This section of the report seeks to answer the following questions: (1) How aware were teachers 
that the CPA had been tailored for their school? (2) Does the tailoring process increase teacher 
commitment, or buy-in? 

How Aware Were Teachers That the CPA Had Been Tailored for Their 
School? 

Teacher surveys and interviews elicited data on teacher awareness of the tailored CPA. From 
responses to the teacher survey, it was determined that 79 percent of teachers in LCR schools and 
100 percent of teachers in HCR schools reported they were aware that the CPA was created or 
tailored specifically for use in each school through teacher collaboration with WestEd staff. In 
LCR schools, 21 percent of the teachers said they were unaware that the CPA was 
created/tailored through such collaboration. Because successful implementation of the CPA 
requires collegial collaboration, this finding suggests serious communication problems in the 
LCR school. 
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Table 2: Percent of teachers who were aware that the portfolio was created/tailored by 
teachers from their school working with WestEd staff 





LCR 


HCR 




(N=38) 


(N=14) 


Yes 


79 


100 


No 


21 


0 



When asked to judge the awareness of other teachers in the school, teachers in both LCR and 
HCR schools generally thought that from some to almost all of the teachers who used the CPA 
knew it was created/tailored for use in their school. In LCR schools, however, 9 percent of 
respondents felt almost none of the other CPA teachers in their school were aware that the CPA 
was tailored. 



Table 3: Percent of teacher responses about the number of other teachers in their school 
who used the CPA in their classes who were aware that the program was 
created/tailored specifically for use in their school 





LCR 


HCR 




(N=35) 


(N=14) 


Almost all 


31 


57 


Many 


29 


0 


Some 


31 


43 


Almost none 


9 


0 



The teacher survey also provided data on the percentage of respondents who participated in the 
tailoring process with WestEd staff. In LCR schools, 57 percent responded in the affirmative; in 
HCR schools, 79 percent responded that they participated in the process. In the schools where 
there was more teacher participation in the tailoring process, there was also more school-wide 
awareness that the portfolio was created or tailored specifically for individual schools. 



Table 4: Percent of teachers who participated in the tailoring process 





LCR 


HCR 




(N=35) 


(N=14) 


Yes 


57 


79 


No 


43 


21 
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Does the Tailoring Process Increase Teacher Buy-in? 

Teachers were asked in interviews about whether the tailoring process mattered to them and to 
other teachers. All interviewees responded positively to this line of questioning. The benefits 
teachers saw for themselves and others were: improved buy-in and ownership, relevance to the 
school and community, and relevance to the program. 

Among the comments pertaining to buy-in were the following: 

• It is veiy important. We are a real different school here. 

• The opportunity to create helps with buy-in. 

• It helped buy-in; it was important we got to choose. 

• If it wasn’t tailored, we would not have done it. We shaped it for our needs. We don’t 
take anything prepackaged. 

• No one would have been happy if we took it the way it was. 

• Our CPA was not generic; it was customized. 

• I would not have been willing to use the CPA in my class without the opportunity to 
change, customize, and streamline it. 

One teacher in an academy-based school where two of four academies were participating in the 
CPA expressed concern about lack of ownership among teachers in the two academies that did 
not participate. “Ownership is a problem,” he commented, “because not everyone is on board. 
The other teachers need to think it is theirs, too.” 

Comments pertaining to relevance of the CPA to the school and community included the 
following: 

• It reflected our community. 

• It was good to use examples from our community. 

• Reflects our community and standards. 

Among comments pertaining to relevance to the program were the following: 

• The process was important; without this opportunity, the program would have flopped. 

• It is veiy important that we have something that works for our site. 

• Had we used samples we saw from other schools, it would not have applied to all my 
students. 

• It was important for me that the CPA was aligned with what we are teaching. 

Teacher interviews asked for opinions about how to improve the tailoring process. More than 
half of the respondents felt there were no improvements necessary. Other respondents 
emphasized the need to begin early in the school year. There were a few comments about the 
need to work out some mechanical issues pertaining to the portfolios, such as finding ways to 
demonstrate the skill of student collaboration, use of multi-media, and the need to conserve 
computer memory. Other comments called for the involvement of more teachers, more 
administrator support, and considering tailoring the CPA for different “tracks,” such as college 
prep and vocational. 
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In summary, the opportunity to tailor the CPA to local needs was very well received. Teachers 
and administrators reported a sense of ownership because the CPA was tailored for their 
particular school and, more importantly, school personnel were included in the tailoring process. 
All of the schools renamed the CPA so that it carried the school name in its title. The data 
indicate that teachers who did not participate in the tailoring process and the accompanying 
professional development did not have as much ownership of the CPA as did those who actually 
worked on tailoring the document. In some schools, the implementation plan called for a core 
group of trained teachers to bring their untrained colleagues on board; this generally was not 
effective. 

Teachers in HCR schools were more likely to feel more aware, more involved, and better 
prepared than did teachers in LCR schools. HCR teachers gave higher marks to the professional 
development they received and the quality of the communication from their leadership than did 
teachers in LCR schools. In addition, teachers in HCR schools reported more personal awareness 
of the tailoring process and more awareness by other teachers that the CPA was tailored 
specifically for their schools than did teachers in the LCR schools. Finally, in the HCR schools, 
there was a greater percentage of teachers who participated in the tailoring process than there was 
in the LCR schools. Involving more teachers in the tailoring process appears to be important to 
increasing teacher commitment or buy-in. 

II. Implementation 

The report seeks to answer the following questions about implementation: (1) What are the key 
conditions for successful implementation of the CPA? (2) Did teachers and students value the 
CPA? (3) How can implementation of the CPA be improved? (4) Did teachers receive 
appropriate support to develop and implement a CPA tailored to their schools’ needs and 
standards? 

What Are the Key Conditions for Successful Implementation of the CPA? 

Prior to presenting data and analyses about this question, it is important to note that 
implementing portfolios school-wide at the high school level, and particularly in large, 
traditionally structured high schools, has proven to be quite difficult. Few high schools can 
generate the teacher buy-in to try, and even those find the curricular and logistical issues 
(communicating the portfolio’s purpose, explaining content and performance standards to 
students, coordinating portfolio tasks across disciplines, professional development, portfolio 
storage, setting up scoring sessions, etc.) daunting. Schools’ and WestEd’s expectation that large 
numbers of completed portfolios could be produced in the first year of implementation were 
demanding. For LCR schools, which lacked many of the structural characteristics that support 
successful portfolio implementation, the expectation was even more demanding. All schools that 
took on the CPA challenge should be commended. Also, all schools learned a great deal about 
implementing portfolios and their capacity to act as a school (versus as a group of departments or 
as a group of teachers). 
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Data pertaining to the implementation questions come from teacher and student surveys, 
interviews with principals/lead teachers, and scorer surveys. The following issues were 
examined: 

• factors that contributed to the success of classroom implementation; 

• value of the portfolio guidelines; and 

• schools’ ability to set up and implement portfolio scoring. 

Factors that contributed to the success of classroom implementation: Teacher surveys 
explored the question by asking respondents to identify what they saw as the five most important 
factors or resources that contributed to the success of the CPA in their classrooms during the 
1998-1999 school year. The survey did not provide a definition of “success,” allowing teachers 
to apply their own definitions. Even if a school was LCR, its teachers still felt that they had 
some success with the portfolios and therefore were able to answer survey questions about 
“success.” Table 5 presents the survey data. 

In LCR schools, the factors most often selected as contributing to the “success” teachers had are 
listed below. (Please note that some schools could not select some of the choices presented by 
the survey. For example, “block scheduling” was a choice in the survey but not all schools had 
block scheduling.) 

• experience with standards (55 percent) 

• school support of standards (52 percent) 

. • significant numbers of students in common (38 percent) 

• student guidelines (36 percent) 

• professional development related to the CPA (36 percent) 

c 

In HCR schools, the “success” factors most often selected were: 

• significant numbers of students in common (65 percent) 

• team teaching (59 percent) 

• integrated curriculum (58 percent) 

• block scheduling (41 percent) 

• career academies (41 percent) 

The data are characterized by their differences between the two groups. With the exception of 
“significant numbers of students in common with other teachers using the CPA,” the selections 
by teachers in the LCR schools were substantially different than those by teachers in the HCR 
schools. This can be explained in part by examining the structure of the four HCR schools. 

They share some common features: School-to-Work focus, integrated curriculum, and significant 
numbers of students in common. Three of the schools used the academy model for most of the 
students implementing the portfolio. The fourth is a small high school in which the entire 
student body participated. In essence, the factors selected by teachers in HCR schools are 
related to the structures of their schools and, in particular, structures that permit and require the 
cooperation of teachers across disciplines for a common cohort of students. Data from teacher 



interviews support this hypothesis and suggest that the structures made implementation 
significantly easier. It must also be noted that the enabling stmctures at the HCR schools are 
mostly present in schools that can be characterized as “reforming.” 

In the LCR schools, there is less uniformity across schools as to stmctures. Two of the schools 
are large comprehensive high schools that attempted to implement the CPA across a single 
cohort of students; e.g., all incoming freshmen. The other two schools are small alternative high 
schools that receive students from and return them to a comprehensive high school throughout 
the school year. In these schools, the entire student body participated. In the four LCR schools 
there was not a significant School-to-Work focus, or integrated curriculum or significant team 
teaching. The responses of teachers in LCR schools instead focus on factors that were within 
their own experiential purview: experience with the standards, school-wide support of the 
standards and the CPA, professional development, and significant numbers of students in 
common with other teachers using the CPA. The five factors selected by teachers in HCR 
schools may not be present in LCR schools and are not likely to be selected as responses by 
teachers in the LCR group. 

Teacher interview data also suggest that the enabling structures support teacher buy-in and that 
that teacher buy-in was influenced by the fact that the CPA was a standards-based tool for 
instmction and assessment. WestEd staff who worked closely with all the schools indicated that 
teacher “buy-in” was an essential success factor and that the HCR schools all had it while the 
LCR schools clearly did not. 



Table 5: Percent of teachers selecting factor as important to success of the CPA in their 
classrooms 





LCR 


HCR 




(N=43) 


(N=19) 


Integrated curriculum 


14 


58 


Previous experience with standards-based assessment 


19 


24 


Previous experience with performance-based assessment 


26 


24 


Block scheduling 


10 


41 


Significant numbers of students in common with other 
teachers using the CPA 


38 


65 


Team teaching 


2 


59 


Career academies 


12 


41 


Professional development related to the CPA 


36 


6 


Class time for the CPA 


12 


18 


Experience with the standards 


55 


12 


CPA student guidelines 


36 


24 


Students with more experience with portfolios 


19 


18 


Parent understanding of portfolios 


5 


6 


School-wide support of the standards 


52 


18 


More school-wide support of the CPA 


31 


12 
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Interviews with teachers and principals/lead teachers asked them to list the three most important 
ingredients to successful implementation. Teacher comments emphasized the importance of: 

• sufficient time and resources 

• leadership and organization 

• administrator and staff commitment 

• tailoring the CPA 

• alignment of instruction and assessment with standards 

Principal/lead teacher comments emphasized the importance of: 

• organization, communication, responsibility 

• teacher commitment and teamwork 

• standards alignment 

• tailoring the CPA 

• time for planning and collaboration 

The interviews reinforce the data from teacher surveys indicating that organization and structure 
relate to cooperation and teamwork. Interviews also suggest that tailoring the CPA was an 
important ingredient to successful implementation because it fostered a sense of ownership 
among teachers. 

Implementation goals and strategies varied across the eight schools. Generally, they fell into 
three categories: (1) whole school, (2) one or two teams or academies within the school, and (3) 
all students within a cohort; e.g., all freshmen. 

Table 6 on the next page lays out some of the characteristics of the implementation data. 
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Table 6: CPA Implementation Data 
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Value of the CPA guidelines: Teachers and students were asked on their respective survey 
instruments whether students received the CPA student guidelines. The guidelines define the 
portfolio’s standards and tasks, and contain the scoring rubric. They are designed to be a stand- 
alone document that provides students with all the information they need to complete the 
portfolio. Teachers created their schools’ guidelines by tailoring the fixed-model CPA 
guidelines. More than 90 percent of teachers in LCR and HCR schools reported that their 
students received a copy of the guidelines. There is a substantial difference bet.ween students in 
LCR and HCR schools regarding student receipt of the student guidelines. In LCR schools, 58 
percent of the students said they received them, while 71 percent of the students in HCR schools 
reported receiving a copy. A possible explanation is that teacher distribution of the guidelines 
was done differently across schools and that some students may not have recognized or 
remembered receiving guidelines. Generally, in HCR schools, teachers gave students their own 
individual copies of the student guidelines booklet prepared by WestEd staff. In LCR schools, 
teachers often gave students excerpts from the guidelines booklet and distributed them from time 
to time throughout the school year. In the student survey, students were asked about the receipt 
of a copy of the “Guidelines for Students,” implying the receipt of a booklet, not a series of 
excerpts. 

Of those students who reported receiving a copy of the guidelines, 62 percent of students in LCR 
schools reported that they were helpful and 85 percent of students in HCR schools felt they were 
helpful. A greater percentage of teachers felt the guidelines were useful than did students. In 
LCR schools, 84 percent of teachers reported that the guidelines were helpful to students, and in 
HCR schools, 100 percent of teachers felt the guidelines were helpful. The discrepancy between 
students in LCR and HCR schools as to the usefulness of the guidelines may be a factor in the 
CPA completion rate. Data reported later in this report suggest a relationship among teacher 
buy-in, student performance, and the value teachers place on the CPA. In HCR schools (where 
completion rates and buy-in are higher than in LCR schools), more teachers place a high value on 
the CPA than do teachers in LCR schools. It is likely that teachers who place a high value on the 
CPA also place a high value on the student guidelines. 



Table 7: Student and teacher reports about CPA student guidelines 

Teachers Students 





LCR 


HCR 


LCR 


HCR 




(N=41) 


(N=16) 


(N=673) 


(N=471) 


Students received guidelines 


93% 


100% 


58% 


71% 


Guidelines were helpful to students 


84% 


100% 


62% 


85% 



School’s ability to set up and implement portfolio scoring: The four HCR schools each had a 
sufficient number of completed portfolios to warrant full scoring; a fifth school fell short of 
being able to conduct scoring, but had a sufficient number of completed portfolios to complete a 
benchmarking activity. This consisted of identifying a small number of portfolios at each score 
level to serve as exemplars. 
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More than 90 percent of respondents who completed scorer surveys felt the content of the 
training and the time devoted to training for scoring/benchmarking the portfolios was adequate, 
and that the experience of participating in scoring/benchmarking was valuable. Moreover, 87 
percent of the respondents expressed confidence that the school will be able to effectively 
conduct scoring/benchmarking next year without assistance/training from WestEd staff. 

Interviews with .principals/lead teachers indicated “no problems” related to setting up the 
sessions; one principal commented, “It was not nearly as difficult to do as people thought it 
might be.” One principal reported some difficulty in obtaining participation from community 
representatives; in another school, this did not pose any problems. 

The ability to conduct scoring is an important component of implementing the CPA; however, an 
added positive factor is that the act of participation in the scoring process expands teacher 
capacity to implement the CPA. Scorers’ surveys indicated that all teachers who participated in 
the scoring process reported they were better prepared to help their students produce portfolios 
that would earn ratings at the highest proficiency level. In sum, scorers and administrators felt 
that scoring was not overly burdensome and increased understanding of their schools’ portfolios. 
However, as will be detailed is the scoring section of this report, significant increases in time 
devoted to scoring appear necessary to increase reliability. 

Did Teachers and Students Value the CPA? 

Data pertaining to this question come from teacher and student surveys and interviews with 
teachers and principals/lead teachers. The following issues were examined: 

• teacher commitment to the CPA; 

• teacher and student perceptions of the benefits of the CPA and the skills it assesses; and 

• teacher and student perceptions of the value of the CPA. 



Teacher commitment to the CPA: Teacher commitment, or “buy-in,” represents a substantial 
component for successful implementation and is borne out in interviews with teachers, as well as 
principals/lead teachers. When asked to offer advice to hypothetical colleagues in other schools 
about necessary ingredients for successful implementation of the CPA, teacher buy-in was a 
recurring theme. 

Teacher surveys indicated that teachers in both LCR and HCR schools felt there was a strong 
commitment to the implementation of the CPA. However, in LCR schools, 90 percent agreed or 
strongly agreed, while in HCR schools, 79 percent agreed or strongly agreed. A possible 
explanation for the difference between teacher reports is the presence of academy-based 
programs in the HCR schools. In some of these schools, the high completion rates represented 
student work in one or two academies; interviews with teachers in these schools indicated that 
teachers in the other academies did not necessarily have a commitment to the CPA. The 
instrument asked respondents to comment on commitment in the school in general, not on a 
subgroup, team, or academy within it. 
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This explanation is supported by the greater percentage of teachers (21 percent) in HCR schools 
who disagree that there is a strong commitment to CPA by teachers and administrators in their 
schools than is the case in LCR schools (10 percent). 



Table 8: Percent of teachers who report strong commitment to CPA by teachers 
and administrators 





LCR 


HCR 




(N=39) 


(N=14) 


Strongly agree 


28 


43 


Agree 


62 


36 


Disagree 


10 


21 


Strongly disagree 


0 


0 



Interviews with teachers suggested that buy-in was related to a cooperative, team approach to 
implementing the CPA. In one academy-based school, a teacher reported that there was a great 
deal of buy-in among teachers in one academy “because we work as a team and discuss the issues 
so we know what we are doing.” In another school, a teacher reported that serving on the 
committee that tailored the portfolio to her particular school was the key to teacher buy-in and 
commitment. 

Teacher and student perceptions of the benefits of the CPA: Teacher commitment to an 
educational intervention is usually determined by the value teachers assign to it in terms of its 
benefits to students. During interviews with teachers, respondents were asked to cite the primary 
purpose of the CPA. The most frequent response was that it benefited students. Teacher and 
student surveys included questions pertaining to the benefits of the CPA and the skills assessed. 

There was general agreement among teacher respondents that the skills upon which the CPA was 
based represented skills valued by both employers and post-secondary institutions. In HCR 
schools, a greater percentage of teachers reported benefits of the CPA in terms of improving 
student work, interest to employers, and interest to post-secondary institutions than did tehchers 
in LCR schools. 

Student surveys indicate that a higher percentage of students in the HCR schools believe that the 
CPA and the skills upon which it is based are beneficial than do students in the LCR schools. 

See Table 9 for the survey data. 
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Table 9: Percent of teachers and students identifying benefits of the CPA and skills 
measured on the CPA 



Teachers Students 





LCR 


HCR 


LCR 


HCR 




(N=41) 


(N=16) 


(673) 


(N=471) 


Employers value skills 


98 


100 


78 


85 


Post-secondary institutions value skills 


98 


100 


78 


84 


CPA helps students master skills 


89 


88 


54 


68 


Skills will help students in the future 


* 


* 


73 


86 


CPA improves students’ academic work 


77 


87 


55 


63 


CPA is of interest to employers 


68 


94 


56 


68 


CPA is of interest to post-secondary institutions 


54 


64 


58 


67 


CPA will help students in the future 


* 


* 


58 


75 


Recommend that other students do the CPA 


* 


* 


59 


78 



*Item was not included on Teacher Survey. 



Teacher and student perceptions of the value of the CPA and the standards assessed by the 
CPA: Teacher survey data indicate general agreement that the CPA is important for students; 
however, teachers in HCR schools are much more likely to value the CPA in terms of its 
importance to teachers than do teachers in LCR schools. Moreover, in LCR schools, teachers 
report that only 41 percent of their students are motivated and interested in the CPA; 86 percent 
of HCR teachers report that students are motivated and interested. 



Table 10: Percent of teachers who value the CPA 





LCR 


HCR 




(N=41) 


(N=16) 


CPA is a valuable assessment for teachers 


68 


94 


Students are motivated and interested in CPA 


41 


86 


CPA is important for students 


87 


88 


Scoring the CPA increased my knowledge of student mastery 


N/A 


98 



Student survey data in Table 1 1 indicate that 91 percent of students in HCR schools believed 
their teachers felt the CPA was important compared to 69 percent of students in LCR schools. 
Also, more than 80 percent of students in schools at both completion rate levels report that their 
teachers value the standards; yet more students in the HCR schools believe the CPA represents a 
good way to teach, learn, and assess the standards than do students in the LCR schools. 
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Table 11: Percent of students who value the CPA and the standards assessed by the CPA 





LCR 


HCR 




(N=673) 


(N=471) 


My teachers think it is important for me to learn the skills 


82 


93 


I understood the skills 


80 


. 91 


My teacher spent enough time explaining the skills 


79 


77 


I know when I have mastered the skills 


63 


78 


My teachers think the CPA is important 


69 


91 


CPA is a good teaching and learning tool 


59 


75 


CPA is a good way to measure skills 


66 


81 


I understood how to do the entries that make up the CPA 


73 


80 



Interviews with teachers and principals/lead teachers elicited comments about the CPA’s impact 
on students. In addition to the feeling that the CPA would help with access to jobs and college, 
teachers and principals/lead teachers believed there was an intrinsic benefit to students. They 
pointed to the pride students took in completing their entries, the sense of achievement they 
gained as they demonstrated mastery of the skills, and the personal growth that occurred as a 
result of the reflective component of the CPA. Some respondents also commented on the stress 
that the portfolio imposed on students, but it was felt that this would be alleviated in the future by 
beginning implementation earlier in the school year. 

Teacher commitment, or buy-in, seems to be present in large measure based on the value placed 
on the CPA. Because the data provide a sample of teachers and students in HCR and LCR 
schools, it is possible to compare the degree to which the CPA is valued in both settings. It 
appears that the importance of the standards or skills are valued highly in both settings; however, 
in LCR schools, the CPA is not valued as highly as in the HCR schools. This again suggests that 
in those schools where a greater percentage of teachers value the CPA, teacher buy-in is greater. 
The teamwork and collaboration required for successful implementation of the CPA also requires 
that a substantial number of teachers believe that the CPA is important. 

How Can the Implementation of the CPA Be Improved? 

Teachers’ surveys explored the question by asking teachers to identify what they saw as the five 
most important factors or resources that would have made the CPA more successful. In other 
words, we wanted teachers to identify the resources they did not have (or have enough of ) but 
thought would have been useful. 

The top five factors for LCR schools were: 

1 . More teacher experience with the CPA (50 percent) 

2. Students with more experience with portfolios (43 percent) 

3. More school-wide support of the CPA (43 percent) 

4. More class time for the CPA (38 percent) 

5. More professional development related to the CPA (31 percent) 







25 



42 



The top six factors for HCR schools were: 

1 . More teacher experience with the CPA (59 percent) 

2. Students with more experience with portfolios (53 percent) 

3. More school-wide support of the CPA (29 percent) 

4. More school-wide support of the standards (29 percent) 

5. More experience with the standards (29 percent) 

6. Integrated curriculum (29 percent) 

The top five factors that overlapped were: more teacher experience with the CPA, students with 
more experience with portfolios, and more school-wide support of the CPA. LCR schools appear 
to have more consensus about their desired factors than did HCR schools; four of the five top 
LCR factors came in at over 35 percent. Two of the top HCR factors came in at over 35 percent. 

The largest differences between LCR and HCR schools occurred in the following factors: 





LCR 


HCR 


More professional development 


31 


6 


Different professional 
development 


19 


0 


More school-wide support of the 
CPA 


43 


29 


Integrated curriculum 


17 


29 



The substantial disparities between LCR and HCR teachers regarding professional development 
and school-wide support may be quite telling. Background about professional development is 
important. All professional development was delivered at each school site by WestEd staff who 
followed a standardized format and distributed the same materials to all participants. 
Furthermore, schools received the same amount of professional development. What varied were 
the number of attendees at the sessions and the conditions under which teachers attended 
(paid/unpaid, mandatory/optional). Regardless of what varied, LCR teachers felt less prepared to 
implement the CPA. They also felt much less supported by their schools. 

Interviews with principals/lead teachers in LCR schools indicated a general level of frustration 
about the inability of their school to implement the CPA at a high level by the end of the school 
year. They suggested possible remedies that reached beyond the classroom to include more 
support, including a need for more or different preparation for implementing the CPA. 
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Table 12: Percent of teachers selecting factors they did not have which would have made 
the CPA more successful in the classroom 





LCR 


HCR 




(N=43) 


(N=19) 


Integrated curriculum 


17 


29 


Previous experience with standards-based assessment 


10 


6 


Previous experience with performance-based assessment 


12 


6 


Block scheduling 


12 


0 


Significant numbers of students in common with other 






teachers using the CPA 


19 


12 


Team teaching 


19 


18 


Career academies 


12 


12 


More professional development related to the CPA 


31 


6 


Different professional development related to the CPA than received 


19 


0 


More class tithe for the CPA 


38 


24 


More experience with the standards 


29 


29 


More experience with the CPA 


50 


59 


Different CPA student guidelines 


10 


12 


Students with more experience with portfolios 


43 


53 


More parent understanding of portfolios 


24 


18 


More school-wide support of the standards 


19 


29 


More school-wide support of the CPA 


43 


29 



Student surveys solicited opinions on how to improve the implementation of the CPA. In the 
LCR schools, “more examples” received the highest number of student responses (40 percent); in 
the HCR schools, the highest number of student responses (52 percent) was “more class time.” 



Table 13: Student opinions on improving CPA implementation 



Make no change 

Devote more time to explaining how to do the CPA 
Allow more class time for students to work on the CPA 
Provide more examples to guide the students’ work 



LCR 


HCR 


(N=673) 


(N=471) 


26 


21 


31 


46 


38 


52 


40 


47 
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In the LCR schools, 66 percent of the students felt they received all the help needed, while in the 
HCR schools, 77 percent of the students reported they received all the help they needed. 
Moreover, in the HCR schools, 22 percent of respondents strongly agreed that they received all 
the help they needed, while in LCR schools, 15 percent of the respondents strongly agreed that 
they received the help they needed. It is evident that in both LCR and HCR schools, a sizable 
number of students reported they did not get all the information, help, and support they needed: 
35 percent in the LCR schools and 23 percent in the HCR schools. 



Table 14: Percent of student responses on receiving all the information, help, and support 
needed for success with the CPA 





LCR 


HCR 




(N=39) 


(N=14) 


Strongly agree 


15 


22 


Agree 


51 


55 


Disagree 


26 


20 


Strongly disagree 


9 


3 



Interviews with teachers solicited their opinions about improving implementation of the CPA. 
The issues identified fell into the following categories: 

• more/better leadership on the following issues: including organization, planning, 
guidelines development 

• the importance of beginning implementation early in the school year 

• portfolio mechanics 

• training (more of it and earlier in the school year) 

In one school where computerization of the CPA was a strong component of the program, some 
of the teachers felt the resolution of certain technical issues would improve implementation. 
Interviews with principals/lead teachers resulted in comments about: 

• training (more of it for more people earlier in the year) 

• materials (need for teacher guidelines, electronic templates, work samples on classroom 
and lab computers) 

• leadership issues (importance of teacher leader, more supervision, oversight, follow- 
through) 

In addition to soliciting comments about improving implementation, interviews with teachers and 
principals/lead teachers asked about barriers to implementation. Among teachers and 
principals/lead teachers, time constraints and competition with other projects was the most cited 
barrier. 
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Teachers also cited student resistance, leadership issues, delays in beginning implementation, and 
logistics and scheduling. The other issues cited by principals/lead teachers focused on curriculum 
issues pertaining to balancing technology applications with needs for reflection, leadership, 
technical issues pertaining to posting portfolios on a web server, student resistance, and lack of 
training. 

In summary, school structure and teacher commitment (buy-in) appear to be key conditions for 
implementation. Buy-in appears to be related to the degree to which teachers value the CPA. 

The data indicate that more experience will help improve implementation in both LCR and HCR 
schools. Teachers wanted more experience with the standards and the CPA — both for 
themselves and for students — and more professional development. Students wanted more class 
time and more examples. However, in LCR schools, structural barriers may hinder teacher buy- 
in and may not permit teachers to get more experience or continued implementation. 

The resolution of leadership issues such as organization, planning, and guidelines development 
were seen as a way to improve implementation. Notably absent, however, were indications that 
changes should be made to the school structure. In HCR schools, the structure focused on a 
whole-school effort and provided support that encouraged implementation. In LCR schools that 
were more traditional, departmental, and teacher-centered, the structure was not able to support 
an effort that required an interdisciplinary approach to instruction and assessment. It may be 
unrealistic to expect that teachers in traditional schools would suggest altering the school 
structure to assist in the implementation of a program that in their view is a classroorn practice. 

Did Teachers Receive Appropriate Support to Develop and Implement a CPA 
Tailored to Their Schools’ Needs and Standards? 

Data pertaining to the question come from teacher surveys, teacher interviews, principal/lead 
teacher interviews, and WestEd progress reports. Data from WestEd progress reports describe 
tailoring activities conducted from July through November of 1998. 

Teacher surveys asked several questions to elicit responses pertaining to the quality of the 
professional development sessions. In the HCR schools, teacher ratings were consistently higher 
in terms of understanding a number of aspects pertaining to the tailoring and implementation of 
the CPA. In Table 15, note the striking difference between teachers in LCR and HCR schools in 
the distribution of “strongly agree” and “agree” responses. In each item, HCR teachers had 
higher “strongly agree” ratings. Overall, more than 90 percent of the HCR teachers reported that 
the workshops prepared them adequately to implement the CPA in their classrooms. Nearly one- 
fourth of LCR teachers reported they were not well prepared to implement the CPA, citing 
difficulties in the following areas: using the portfolio rubric, designing assignments which help 
students demonstrate the standards, and integrating the portfolio into curriculum plans. 
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Table 15: Percent of teachers who reported understanding speciflc aspects of the tailored 
CPA after attending CPA professional development workshops 





LCR 


HCR 




(N=39) 


(N= 


15) 


After the professional development workshops, 
I understood: 


SA 


A 


. SA 


A 


The purposes of the portfolio 


27 


63 


67 


33 


The standards 


37 


51 


60 


40 


The entries in the portfolio 
The nature of tasks that elicit work 


21 


67 


47 


53 


demonstrating the standards 
How to judge when work demonstrates 


18 


69 


40 


60 


the standards 


20 


70 


33 


60 


How to use the portfolio rubric 
How to design assignments which help 


22 


57 


47 


47 


students demonstrate the standards 
How to integrate the portfolio into my 


19 


54 . 


40 


53 


curriculum plans 

Overall, the workshops prepared me 

adequately to implement the portfolio 


16 


58 


33 


60 


in my classes 


22 


57 


27 


67 



The teacher survey asked respondents to rate the professional development activities in terms of 
the amount of time spent and the quality of the training. These data are in Table 16. More than 
50 percent of respondents in the LCR schools felt they would have been better prepared had more 
time been devoted to professional development from WestEd. One third of the HCR teachers 
reported they would have liked more time for professional development. 

Interviews with teachers and principals/lead teachers indicated that time allocations for 
professional development are always far less than the perceived need. It is a struggle that goes on 
in every school district, trying to balance need with available time, weighing the value of 
summer, Saturday, and evening sessions. The high levels of satisfaction reported by HCR and 
LCR teachers should be considered within the context that “not enough time” is the usual order 
of business. 

In terms of quality, 64 percent of teachers in LCR schools and 73 percent in HCR schools 
appeared to be completely satisfied with the quality of professional development provided by 
WestEd. 
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Table 16: Teacher ratings about the adequacy of time and quality of professional 
development 

LCR HCR 

(N=39) (N=15) 

SA A D SD SA A D 


SD 


I could have been better prepared 
to implement the tailored CPA if: 














More time had been 
devoted to professional 
development by WestEd 


5 


54 21 


21 


7 


27 60 


7 


The professional development 
from WestEd had been of 
higher quality 


8 


28 42 


22 


0 


27 60 


13 



Teachers were asked during interviews about ways to improve the professional development 
provided by WestEd. Half of the respondents cited issues that can be grouped under a category 
called “leadership.” This includes communicating expectations to teachers and students, 
communicating rewards and consequences, conducting more follow-up. Many respondents 
emphasized the importance of providing training early in the school year, even during the 
summer preceding portfolio implementation. Half of the teachers interviewed had no 
suggestions for improving the professional, development; one respondent summed it up by 
saying, “Leave it the way it is. WestEd did a great job.” 

The teacher surveys provide strong evidence that the school leadership or teacher leadership did 
support the professional development sessions by clearly communicating the purpose of the 
tailored CPA, particularly in HCR schools. In LCR schools, only 14 percent of respondents 
reported the leadership did not communicate the purpose of the CPA adequately. Again, note the 
striking difference between teachers in LCR and HCR schools in the distribution of “strongly 
agree” and “agree” responses. 



Table 17: Percent of teachers who strongly agreed or agreed that school administration or 
teacher leadership communicated the purpose of the CPA clearly 



LCR 

(N=40) 

SA A 

23 63 



HCR 

(N=15) 

SA A 

60 40 
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III. Student Performance and Assessment 



This section of the report seeks to answer the questions (1) How reliable were the tailored 
portfolio as assessments?, (2) What is the relationship between dimensional and overall scores?, 
and (3) How well did students perform? 

The primary data sources are the results of the scoring of 287 portfolios at four schools in the 
spring of 1999 and questionnaires from scorers. While the portfolio results can help answer the 
three questions, the conditions under which scoring occurred make it possible to give answers 
with important caveats. As will be detailed in the following pages, portfolios were scored under 
circumstances that reflect the reality that schools had difficulty in recruiting and training 
qualified scorers for the time necessary to score with high reliability . Therefore, it is 
unreasonable to expect highly reliable scores from the schools examined in this report. That said, 
the first question (about reliability) demands an answer so that schools can make judgements 
about the appropriateness of the assessment for high-stakes use and the conditions necessary to 
obtain reliable results. The second question relates to the first and drives at reliability from the 
dimensional perspective. If we are able to identify dimensions which are particularly difficult to 
measure reliably, it might be appropriate to alter or remove the dimension from the model 
scoring rubric. Also, practitioners might use this information in tailoring their rubrics. The third 
question is essential because practitioners need to know under what circumstances students can 
perform adequately and what distribution of scores they might expect. For example, some 
schools might conclude that the percentage of Advanced portfolios is generally too low, and 
therefore that they need to alter their curriculum to elicit more Advanced scores. Others might 
want to reexamine their definition of Advanced. 

Prior to presenting scoring data it is appropriate to describe the scoring process that yielded the 
data. Each school scored its own portfolios following limited training by WestEd staff. 

The Scoring Process 

Scorers: At every site scorers included educators who had been involved in the portfolio 
process, but each school had additional scorers. The categories from which additional scorers 
were selected included educators from the school who were not involved in the portfolio process, 
school or district administrators, employers, parents, and school board members. Table 18 shows 
the proportions of scorers by type across the four schools. No distinction was made on the 
questionnaire to distinguish teachers involved with the portfolio from those who were not. As 
Table 18 below shows, teachers were the large plurality of scorers (44 percent) and a majority 
were educators (teachers or administrators, percent). The remaining types of scorers were 
relatively evenly distributed across four other types. 
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Table 18: Scorers by type 



Category of scorer 


Drake 


New 

Technology 


Peoria 


San Leandro 


Teacher 


5 


2 


21 


. 3 




(36%) 


(10%) 


(84%) 


(27%) 


School or district 




1 


4 


4 


administrator 




(5%) 


(6%) 


(36%) 


Employers 




7 










(33%) 






Parents 


8 


1 








(57%) 


(5%) 






Community leaders 


1 


3 




2 




(7%) 


(14%) 




(18%) 


Other 




7 




2 






(33%) 




(18%) 


Totals 


14 


21 


25 


11 



Anchor pulling: Prior to the scoring session, WestEd staff facilitated an anchor-pulling session 
with a small group of teachers (between one and five) involved in the portfolio process. At the 
session, WestEd facilitated the selection of anchor portfolios to exemplify the different 
performance levels on the scoring rubrics and some example portfolios to be used in training 
scorers. Please see Appendix B for the CPA rubric, from which schools tailored their own 
rubrics. 

At each school, nine portfolios were selected (an anchor and two training portfolios for each of 
the three performance levels on the scoring guide — Basic, Proficient, Advanced). In an ideal 
anchor-pulling session there would be ample portfolios to select from and a full range of 
performances amongst portfolios. In the four anchor-pulling sessions there were between 64 and 
91 portfolios from which to select the anchors and training portfolios, a restricted pool in every 
case. This means that it was harder to find portfolios that are clear cut examples of each 
performance level. Without clear examples it is hard for scorers in training to see the distinct 
differences between performance levels. 

During the anchor pulling, scorers developed rationales explaining why a particular portfolio 
exemplified a particular performance level. Scorers presented these rationales during training at 
the actual scoring session. If in the process of anchor-pulling there was good reason to change 
the scoring rubric prior to the scoring session, that was done. This occurred rarely and changes 
were minimal. 
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Training of scorers: In an ideal scoring session, scorers would have a lengthy training followed 
by a series of pre-scored “calibration” portfolios that they would have to score accurately before 
being allowed to proceed to independent scoring. Then scorers would periodically stop scoring 
and recalibrate to ensure that their interpretation of the rubric had not “drifted.” This two-step 
process ensures that only scorers who are interpreting the rubric in a consistent way are allowed 
to score and improves scoring reliability. The reality for the tailored CPA scoring sessions was 
• different. First, the total length of time that schools arranged for scoring was only one day, often 
ending in mid-afternoon because teachers’ working days normally end then. Also, schools who 
had community members, such as a parents or business leaders, score felt that they were 
impinging on scorers’ time if they kept scorers for a long day. Therefore, lengthy and 
appropriate training was not possible. In addition to limited training time, it was also not feasible 
to calibrate scorers. The reason for this is that, under such schemes, there will almost certainly 
be scorers who do not calibrate and are not allowed to participate in scoring. So, a teacher who 
had been a key person in leading students to develop portfolios might not calibrate. Or, some of 
those community members who volunteered their time might not calibrate. Expelling scorers 
who did not calibrate would often be politically unacceptable in a school setting and would cause 
ill feeling that could undermine the whole portfolio process. 

The scoring sessions began with a brief introduction to the portfolio and the scoring process. 

Next, scorers were introduced to the scoring rubric and were given an opportunity to read it and 
ask questions. Then, the anchor portfolios were introduced by having scorers read the Advanced 
portfolio first, and then a teacher from the anchor-pulling team gave an explanation of how the 
scoring guide indicates the score that the portfolio received. They gave specific examples of 
student work in the portfolio that matched to the rubric and answered any questions that scorers 
raised. This process was repeated for the Proficient and Basic score points. 

Once scorers were familiar with the scoring rubric and the anchors, they were given a set of three 
practice portfolios. Each scorer independently assigned a score to each portfolio using the 
scoring rubric and anchors to guide their decisions. Then a teacher from the anchor-pulling team 
revealed the score that had been assigned by the anchor-pulling team. They then led the group in 
a discussion about why the portfolio was assigned that score. This process was repeated for the 
other two portfolios. In most cases training took two hours. 

In sum, these conditions naturally led to lower-than-desired reliability . (Schools implementing 
tailored portfolios in the 1999-2000 school year scored under better conditions than in 1998-1999 
and generally achieved higher reliability.) 

Scoring: At Drake, Peoria and San Leandro, portfolios received two scores each, and in some 
cases a third scorer was used to resolve discrepancies between the first two scores. The scoring 
process at New Technology was necessarily different, since all portfolios were electronic and 
viewable online only. Thirty-two percent of the portfolios only received a single score at that site 
because it was harder to track the online scoring process. Of the remaining portfolios, 25 percent 
received two scores and 43 percent received three scores. 
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Performance levels: Each school used three levels of performance to give scores on the 
dimensions of the rubric and an overall score. Table 19 shows the levels of performance. 
Although they have different labels, conceptually these levels were regarded as the same as 
shown by the common interpretation that was in parentheses on all the scoring guides except that 
of New Technology. 



Table 19: Name of performance levels, by school 





Performance Levels 

1 2 3 




(Not ready to show to 
college or employer) 


(Ready to show to 
college or employer) 


(Superior quality, may 
exceed expectations of 
college or employer) 


Drake 


Emerging 


Competent 


Outstanding 


New Tech 


Basic 


Proficient 


Advanced 


Peoria 


Developing 


Proficient 


Advanced 


San Leandro 


Basic 


Proficient 


Advanced 



Rubric dimensions: Because the portfolio guidelines and scoring rubrics had been tailored to 
the assessment priorities of each school, there was some variation in the dimensions of student 
performance being assessed. Appendix C contains the scoring rubrics used by the scorers at each 
school. 

Table 20 shows the dimensions for each school. The number of dimensions ranged from four (as 
in the original CPA portfolio) to seven. The explanation for this was that the schools with seven 
dimensions deliberately had one dimension for each standard they were assessing, whereas those 
schools with four dimensions collapsed some of their standards into one dimension. 

Table 20: Scoring dimensions assessed, by school 





1 


2 


3 


Dimensions 

4 5 


6 


7 


Drake 


Professional 


Work 


Technology 


Communicati 










Preparation 


Planning 


Skills 


on 








New Tech 


Technology 


Citizenship 


Critical 


Career 


Collabor- 


Written 


Oral Commun- 




Literacy 


& Ethics 


Thinking 


Preparation 


ation 


Commun- 


ication 














ication 




Peoria 


Personal 


Inter- 


Thinking & 


Communicati 


Employment 


Technology 


Making 






personal 


Problem- 


on 


Literacy 


Literacy 


Connections 








solving 










San 


Career 


Analysis 


Technology 


Commun- 








Leandro 


Preparation 






ication 









Across schools there were four dimensions that WestEd staff judged to be equivalent, although 
their titles varied. Table 21 shows these common dimensions. Inter-rater reliability analyses 
treated these as common dimensions. 



Table 21: Common dimensions assessed across schools 





Common Dimensions 
Drake New Tech 


Peoria 


San 

Leandro 


Career 

Preparation 


1 


4 


5 


1 


Thinking and 


2 


3 


3 


2 



Problem- 

Solving 



Technology 


3 


1 


6 


3 


Communication 


4 


6 


4 


4 



Assigned scores: Scorers were instructed to assign a score to every dimension and also to assign 
an overall score that reflected their judgment about the portfolio in its entirety (holistically). 
Scores were assigned by checking a box on the appropriate cell of the grid that corresponded to 
the level of performance for that dimension or overall performance. Scorers could also add a 
plus or a minus to any score, although they were not required to do so. This allowed a scorer who 
was trying to decide between two performance levels a way to assign a score but still reflect that 
it might, for example, be just barely in the Proficient level, or almost in the Advanced level. 
Scores reported to students, however, showed only the main level they were judged to be in for 
the overall score. So, a student was told only that they were in either the 
Basic/emerging/developing, Proficient/competent or Advanced/outstanding level. 

How reliable were the tailored portfolio as assessments? 

Inter-rater reliability is a measure of how often the first independent score assigned to a portfolio 
agreed with the second independent rating given by another scorer to the same portfolio. Two 
types of reliability analyses, inter-rater reliability, and Cronbach’s alpha were performed. To see 
the relationships between the dimensions of the scoring rubrics within schools and across schools 
on the common dimensions, bivariate correlations ancj. multiple regression analyses were 
performed. 

In our analyses, inter-rater reliability was calculated in two ways: first, in terms of exact 
agreement within a scoring level and, second, allowing agreement across main performance 
levels by use of the plus and minus scores. Under the first method, scores were counted as 
agreeing if they were in the same performance level. For example, scores of Proficient minus 
and Proficient plus would count as agreement, but Basic plus and Proficient minus would not , 
even though Basic plus and Proficient minus are “closer” together than Proficient minus and 
Proficient plus. 



Under the second method, scores with exact agreements and scores of plus on one level and 
minus on the level above were regarded as agreeing. For example, Basic plus and Proficient 
minus were counted as agreeing, but Basic plus and Proficient were not. Table 22 shows the 
inter-rater reliabilities for the overall portfolio (versus dimensional) scores under each method, 
and the range of inter-rater reliabilities calculated for individual scorers. The reason for using 
two methods was to see if the use of plus and minus scores improved the agreement of scorers 
when scorers were divided across performance levels, but close (e.g., Basic plu§ and Proficient 
minus). 



Table 22: Inter-rater reliability of overall scores 





Inter-rater Reliability 
(method one, agreement 
within a level) 


Inter-rater Reliability 
(method two, agreement 
within a plus or minus) 


School 


Overall 

% 


Min/Max 
IRR per 
scorer 
% 


Overall 

% 


Min/Max 
IRR per 
scorer 
% 


Drake 


57.89 


33.3 /83.3 


66.67 


33.3 / 83.3 


New Technology 


68.38 


38.9 / 84.2 


73.5 


39.5/85.8 


Peoria 


69.51 


25.0/ 100.0 


75.61 


25.0/ 100.0 


San Leandro 


51.47 


30.8 / 100.0 


52.94 


30.8/ 100.0 



Inter-rater reliabilities were not as high as desirable but were understandable given the difficult 
scoring conditions. Using method two where score agreements are allowed across the level 
boundaries produced slightly higher reliabilities. 

Using method one inter-rater reliabilities ranged from 57.89 percent to 69.51 percent. Using 
method two they ranged from 52.94 percent to 75.61 percent. New Technology and Peoria both 
achieved inter-rater reliability over 73 percent using method two. It is not clear why New 
Technology and Peoria achieved higher levels of reliability. 

Reliabilities for individual scorers varied greatly, as opposed to for scorers at a given school 
taken as a group. Individual inter-rater reliability ranged between 25 percent (the first scorer 
agreeing with the second scorer 25 percent of the time) and 100 percent. There seems to be no 
connection between the wide range of scorer reliability and the overall reliability. This indicates 
that there were unreliable scorers in the sites with higher overall reliability and reliable scorers at 
the sites where overall reliability was low. Furthermore, there was not a clear correlation between 
having a large proportion of teachers as scorers, who might be expected to have a greater shared 
understanding of the rubric than employers, parents, or community members, and higher 
reliability. 





Inter-rater reliability was also calculated for individual dimensions of the scoring guide. These 
are shown below in Table 23. On the whole the reliabilities by dimension were similar to the 
reliabilities of the overall scores. There were some exceptions. In the case of Peoria the 
Professional Preparation had reliability 18.26 percentage points less than that for the overall 
scores. Similarly, the reliability in Thinking and Problem-Solving dimension at New Technology 
was 10.86 percentage points lower than that for the overall score. At San Leandro the 
Communication reliability was 8.19 percentage points lower. There appeared to be no pattern 
across sites about which dimension had lower reliabilities. 



Table 23: Inter-rater reliabilities by dimension 
Method one 



Common 

Dimensions 


Professional 

Preparation 


Thinking and 
Problem-Solving 


Technology Skills 


Communication 


Drake 


58.93% 


51.79% 


58.18% 


58.49% 


New Tech 


58.62% 


57.52% 


69.03% 


60.53% 


Peoria 


51.25% 


57.14% 


64.63% 


57.32% 


San Leandro 


64.71% 


55.88% 


47.06% 


43.28% 


Combined 


58.13% 


55.41% 


61.32% 


55.70% 



Site 

Specific 

Dimensions 


Citizenship 
& Ethics 


Collaboration 


Oral 

Communication 


Personal 


Interpersonal 


Making 

Connections 


New Tech 


52.83% 


58.26% 


57.84% 


NA 


NA 


NA 


Peoria 


NA 


NA 


NA 


53.95% 


61.25% 


58.97% 




38 




Method two 



Common 

Dimensions 


Professional 

Preparation 


Thinking and 
Problem- 
Solving 


Technology 

Skills 


Communication 


Drake 


60.71% 


51.79% 


58.18% 


58.49% 


New Tech 


63.79% 


58.41% 


72.57% 


63.16% 


Peoria 


52.50% 


61.04% 


65.85% 


58.54% 


San Leandro 


64.71% 


55.88% 


47.06% 


' 44.78% 


Combined 


60.63% 


57.32% 


62.89% 


57.28% 



Site Specific 
Dimensions 


Citizenship 
& Ethics 


Collaboration 


Oral 

Communication 


Personal 


Interpersonal 


Making 

Connections 


New Tech 


55.66% 


60.00% 


58.82% 


NA 


NA 


NA 


Peoria 


NA 


NA 


NA 


53.95% 


65.00% 


61.54% 



Test Reliabilities 

The reliability of a test is a statistical calculation of how much consistency exists among the 
measures (items) in the test. Reliability increases as the number of items increases. Ideally, test 
reliability coefficients are obtained by correlating the results of two measurements using the same 
test. When a test is only administered once, another way to obtain the reliability coefficient is to 
split the test results into two halves and then correlate one half of the test with the other half. 

This is known as the split-half method and produces a measure of internal consistency. A more 
sophisticated calculation of the reliability coefficient is to use Cronbach’s alpha (a) which 
produces, in effect, the mean of all possible split-half reliability coefficients. The closer the test 
reliability is to 1, the more consistent it is. 

Reliability coefficients were calculated in three ways: using the scores on all dimensions given in 
the first scoring round, using the scores on all dimensions given in the second scoring round, and 
using both first and second round overall scores only. The reliabilities are shown in Table 24. 




OO 



39 



Table 24: Reliability coefficients for each portfolio 



School 


Reliabili 

First scores on all 
dimensions 


ty Coefficients (Cronl 

Second scores on all 
dimensions 


)ach’s a) 

First and second 
scores for overall 
scores only 


Drake 


.89 


.87 


. -75 


New Technology 


.88 


.89 


.77 


Peoria 


.92 


.94 


.82 


San Leandro 


.93 


.86 


.41 



The reliabilities for first and second scores using all dimensions are reasonable considering this 
was the first year of administration, although the number of scores upon which this is based is 
relatively low. This shows that the portfolio assessment was consistent in measuring the ability 
of students across the dimensions . Reliabilities based on only the first and second round overall 
scores are weaker. This is because they are calculated on only two scores, whereas the dimension 
scores number between 4 and 7, which helps reliability. This indicates the value of scoring by 
dimensions and the accuracy that is lost when simply reporting overall scores . 

Based only on overall scores, the reliability for Peoria is reasonable, for Drake and New 
Technology it is lower than ideal, and for San Leandro it is outside of the acceptable range. This 
may indicate that scorers at San Leandro were making the judgment about overall score in some 
way different than their judgments about scores on dimensions, as the scores by dimension 
produce much higher reliabilities than just the overall scores. This, again, points to the need to 
improve scorer training. 

As a whole, reliability data suggest that improving scoring conditions and training is necessary 
for the portfolio to be used in high-stakes environments. However, it is likely that via increased 
training time and other steps to be mentioned in the next section of this report, reliability could 
increase so that high-stakes use of the portfolio is appropriate, as part of a multiple-measure 
system. 



Critically, current levels of reliability may not significantly impact scalability — no 1998-1999 
schools indicated that lower than desired reliability would decrease implementation. In fact. 
HCR schools are generally expanding implementation . Data from 1999-2000 scoring, where 
emphasis on training and calibration was stronger than in 1998-1999, sheds more light on the 
portfolio's capacity for reliable scoring. In the 1999-2000 effort, where WestEd staff focused on 
increasing training time and better overall scoring conditions, reliability was higher than in 1998- 
1 999 (a range of 64 percent to 79 percent versus a range of 58 percent to 7 1 percent). WestEd 
staff more aggressively communicated the need for reliability to participating schools in 1999- 
2000. Also, in 1999-2000 WestEd staff targeted training at scoring dimensions where it was low 
in 1998-1999 (e.g.. Technology, Analysis). 
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Increasing reliability 



While no definitive data exist about why reliability was lower than desired, or why it varied 
across schools, WestEd staff hypothesize that adhering to the scoring procedures listed below 
would substantially increase reliability. As mentioned previously, these procedures were 
followed in 1999-2000 and reliability generally increased. 

• Longer training (more than the two hours generally used by the 1998-1999 s'chools) is needed 
to ensure that scorers apply the scoring rubric more consistently. In particular, scorers who in 
practice portfolios show low reliability should be targeted for extra training. 

• Target training to dimensions of the rubric that had the lowest inter-rater reliabilities. 

• Examine the wording of dimensions of the rubric that had the lowest inter-rater reliabilities 
and rewrite where appropriate. 



In high-stakes scoring situations, formal calibration where scorers have to get exact agreement on 
a certain percentage of the practice portfolios they scored in order for their scores to count, 
should be considered. Scorers who do not get exact agreement on a certain percentage of the 
practice portfolios would not be allowed to continue or would have their scores not count. Given 
the substantial range in reliability at the level of the individual scorer (32 percent to 92 percent on 
average), calibration could improve reliability considerably. However, if calibration is used, it 
might result in a teacher who had been involved in portfolio development and implementation 
not being allowed to score. 



What is The Relationship Between Dimensional and Overall Scores? 

To investigate the relationship of dimension scores to overall scores, bivariate correlations were 
calculated. The correlations show the strength of the relationship between the scores students 
obtained on each dimension and the overall scores they received for the whole portfolio. The 
closer a correlation is to 1, the stronger the relationship. One would expect the correlation to be 
high, given that the overall score is a summary of how the student did across all dimensions. If it 
is low, then there may be little relationship between what the dimension is measuring and what 
scorers valued in the portfolio overall. Table 25 shows the results of the bivariate correlations. 
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Table 25: Bivariate correlation of dimensions scores and overall scores 



All Sites 




Thinking 


0.805 


Professional Prep 


0.739 


Communication 


0.782 


Technology Skill 


0.706 


Drake 




Thinking 


0.867 


Professional Prep 


0.847 


Communication 


0.856 


Technology Skill 


0.724 


New Technology 




Thinking 


0.814 


Professional Prep 


0.779 


Communication 


0.851 


Technology Skill 


0.693 


Oral 


0.818 


Citizen 


0.673 


Collaboration 


0.734 


Peoria 




Thinking 


0.815 


Professional Prep 


0.665 


Communication 


0.820 


Technology Skill 


0.694 


Connect 


0.879 


Personal 


0.731 


Interpersonal 


0.682 


San Leandro 




Thinking 


0.901 


Professional Prep 


0.751 


Communication 


0.875 


Technology Skill 


0.788 



The correlations show that, for the common dimensions across schools (Thinking and Problem- 
Solving, Professional Preparation, Communication and Technology skills) the relationships of 
the dimensions to the overall score ranges from .706 for Technology to .805 for Thinking and 
Problem-Solving, all of which are fairly strong correlations. This means that there is a strong 
relationship between what was measured in those four dimensions and what was valued by 
scorers in the overall scores. 
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At Drake, the correlations of the dimensions with overall scores are strong. At New Technology, 
there were strong correlations between overall score and Thinking and Problem-Solving, Written 
Communication, and Oral Communication. There were moderate correlations between overall 
score and Professional Preparation and Collaboration, with lower correlations for Technology 
and Citizenship. At Peoria, three of the dimensions (Thinking and Problem-Solving, Making 
Connections, and Communication) had strong correlations with overall score. Overall score was 
moderately correlated with Personal Skills, Technology Skills, Interpersonal Skills and, lastly. 
Professional Preparation. At San Leandro there were strong correlations between overall score 
and the Thinking and Problem-Solving and Communication dimensions, and moderately strong 
correlations with the Technology Skills and Professional Preparation dimensions. 

The Thinking and Problem-Solving dimension consistently correlated strongly with overall 
scores. This is not surprising since every school’s portfolio contained two work samples that 
showed evidence of Thinking and Problem-Solving. Next, the Professional Preparation 
dimension was consistently important as was the Communication dimension. Again, Professional 
Preparation was being assessed in several portfolio entries (the personal statement, resume and 
application forms mainly), and Written Communication was evidenced throughout because the 
portfolios were predominantly text-based. 

Table 26 shows, for each school, a matrix of the correlations between the dimension scores 
calculated just using one set of scores per school. The table shows that the correlations between 
dimensions varied between moderate at New Technology HS to fairly strong at San Leandro HS. 

These relationships indicate that the dinaensions are moderately related to each other and strongly 
related to the total score. It may be that the dimensions are hard to distinguish because the 
portfolio is unlike a traditional assessment in the sense that there are no identifiable items. So, 
instead of the student receiving a separate score for each piece in his/her portfolio, scores are 
based on a judgment of whether a standard is demonstrated anywhere in the portfolio. This 
means that the "items" are really conceptual constructs that might be interpreted differently 
across scorers. Even if all scorers understand the construct, they are simultaneously examining 
the portfolio entries to assign between four and seven different scores. It may be this that is 
contributing to the low inter-rater reliabilities. A way to avoid this would be to give separate 
scores for the different entries in a portfolio, although this seems against the notion of viewing 
the portfolio as a whole. Alternatively, being more directive about where evidence of different 
dimensions might be found could alleviate the problem. For example, “Personal Skills” might 
best be observed in the personal statement, the resume and the letter of recommendation, while 
the best place to see “Thinking and Problem-solving Skills” might be the work samples. 
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Table 26: Correlation matrices of dimensions against dimensions 

Highest and lowest correlations are show in bold for each school. 



Correlation Matrix - Drake HS 





DIMl 


DIM2 


DIM3 


DIM4 


DIMl 


1.0000 








DIM2 


.7984 


1.0000 






DIM3 


.5976 


.6985 


1.0000 




DIM4 


.6528 


.6768 


.5989 


1.0000 



Correlation Matrix - New Technology HS 





DIMl 


DIM2 


DIM3 


DIM4 


DIM5 


DIM6 


DIM7 


DIMl 


1.0000 














DIM2 


.4345 


1.0000 












DIM3 


.4692 


.4663 


1.0000 










DIM4 


.5115 


.3260 


.6848 


1.0000 








DIM5 


.3245 


.3589 


.4061 


.4991 


1.0000 






DIM6 


.5814 


.3888 


.7295 


.6001 


.4042 


1.0000 




DIM? 


.5273 


.3608 


.5336 


.6268 


.6730 


.5883 


1.0000 





Correlation Matrix 


- Peoria 








DIMl 


DIM2 


DIM3 


DIM4 


DIM5 


DIM6 


DIM7 


DIMl 


1.0000 














DIM2 


.7075 


1.0000 












DIM3 


.6185 


.6030 


1.0000 










DIM4 


.6457 


.6677 


.8040 


1.0000 








DIM5 


.6499 


.6135 


.6878 


.7238 


1.0000 






DIM6 


.5374 


.5590 


.7104 


.6261 


.6896 


1.0000 




DIM? 


.6713 


.6721 


.7863 


.7966 


.6803 


.6834 


1.000( 



Correlation Matrix - San Leandro HS 





DIMl 


DIM2 


DIM3 


DIM4 


DIMl 


1.0000 








DIM2 


.7840 


1.0000 






DIM3 


.7312 


.7969 


1.0000 




DIM4 


.7341 


.8174 


.7646 


1.0000 
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Relationship of Overall Scores to Completion Rate 

The overall scores for first and second scores combined were correlated against the number of 
completed entries by student. Please note that the results reflect a truncated sample — only 
portfolios with enough entries to be judged “complete” were analyzed. “Incomplete” portfolios 
were not scored. Table 27 shows the correlations. 

Table 27: Correlation of the overall score with number of completed portfolio entries 



School 


Correlation 


Drake 


.27 


New Technology 


.15 


Peoria 


.22 


San Leandro 


.35 



The correlations are low, showing that there was little relationship between numbers of 
completed entries and the overall quality of the portfolio. This should not be taken to mean that 
it does not matter how many of the portfolio entries were completed by a student. Rather, it 
reflects the fact that, in this first year, not all schools were able to get students to the point of 
completion in time for scoring. Portfolios that had significant omissions, such as no work 
samples, were not scored. Work samples play an important role in showing the student’s ability 
across several of the dimensions that correlated highly with overall score. So, scorers were able 
to make a judgment when some parts of a portfolio were missing, but this does not mean it is a 
desirable situation. After the first year of implementation, schools might choose not to score a 
portfolio that was missing entries. 

How Well Did Students Perform? 

Table 28 shows the distribution of scores across the performance levels. In the simplest • 
assessment of student performance, 68-percent of students who completed portfolios “passed.” or 
received a Proficient or Advanced score, and 32 percent “failed” or received a Basic . In three of 
the schools (Drake, New Technology and San Leandro) the majority of scores fall within the 
Proficient category, but at Peoria, most portfolios were judged to be Basic. At Drake, 26.4 
percent of the portfolios were scored as Advanced. 

When considering the strength of these scores, the reader should know that only “complete” 
portfolios (versus incomplete or portfolios where not all entries were done) were scored. This is 
important as quality correlates with completeness — students with the skills and attitudes to 
complete a portfolio typically submit higher quality entries than those that do not complete 
portfolios. 

The distribution of scores across the four schools was fairly similar, with the exception of Peoria. 
(Peoria had 58.9 percent Basic as opposed to an average of 23.2 percent for the other three 
schools.) Excluding Peoria, the range of Basic was 17.4 percent to 30.2 percent; Proficient 5 1 .8 
percent to 58. 1 percent; and Advanced 1 1 .8 percent to 26.4 percent. No reason for Peoria’s 
lower scores is apparent. 
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In comparison to the 1996-1997 cohort, the overall percentage of Proficient scores was the same 
at 50 percent. Advanced was similar (18 percent in 98-99 and 12 percent in 96-97). At the Basic 
level, percentages were also similar (32 percent in 98-99 and 38 percent in 96-97). 

While the number of schools, students, and portfolios participating in 1996-1997 and 1998-1999 
is not large, data suggest that schools possessing the factors for successful implementation 
identified in 1996-1997 and confirmed in 1998-1999 might expect student performance to fall 
within the ranges seen in 1996-1997 and 1998-1999 . Furthermore, schools might use these 
expected ranges to decide if they would like to go to a four-point scale to alter the percentage at a 
given performance level. They might also use these ranges, in combination with reliability data, 
to determine what stakes to associate with performance. 



Table 28: Score distributions per school 





Score Assigned 


School 


Basic 

% 


Proficient 

% 


Advanced 

% 


Drake score 1 


21.0 


47.3 


31.7 


score 2 


22.8 


56.2 


21.0 


imedn ; s 








New Technology score 1 


19.5 


62.4 


18.2 


score 2 


15.3 


51.9 


32.6 


nie^ j 




. ,57.15 ■ - 


-^25, 4::r". 


Peoria score 1 


63.8 


29.7 


6.6 


score 2 


53.9 


37.4 


8.8 


me^ii 


58.9 


■33.6: -. ' ' ^ 


r 7.7 


San Leandro score 1 


32.3 


55.9 


11.8 


score 2 


28.0 


60.3 


11.8 




30.2 


; :■■■ :58.1^' ■. 


11.8 


Overall 




■■li0.2;%,: 
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IV. Whole-school change 



This section of the report seeks to answer the following questions: (1) What is the impact of the 
tailored CPA on teachers and school practices? (2) What is the impact of the tailored CPA on 
teacher understanding of standards-based teaching, assessment, and portfolios? (3) What is the 
impact of the tailored CPA on whole-school change, particularly in schools with traditional 
structures? The great majority of the data analyzed in this section comes from mterviews and 
surveys. It is likely that some response bias favorable to the CPA exists in these data — teachers 
who responded to the survey may be disproportionately inclined toward positive views of the 
CPA. 

What is the Impact of the Tailored CPA on Teacher and School Practices? 

Data pertaining to the question come from teacher and student surveys, and interviews with 
teachers and principals/lead teachers. The following issues were examined: (a) impact on 
teacher practices and (b) use of scoring data in the classroom and the school. 

a. Impact on teacher practices . This issue was examined by several items on the teacher survey 
that asked teachers to rate the degree to which they altered their classroom practices in response 
to the standards and the CPA. A related question focused on the degree to which teachers were 
able to structure their curriculum. In the LCR schools, 77 percent of the respondents reported 
they were able to structure their curriculum as they chose; in the HCR schools, 88 percent of the 
teachers reported they were able to do so. 



Table 29: Percent of teachers who reported ability to structure curriculum 



In this school, I am able to 
structure my curriculum as I choose. 



LCR HCR 



(N=40) 


(N=16) 


Yes 


No 


Yes 


No 


77 


23 


88 


13 



More than 70 percent of teachers in both LCR and HCR schools reported they engaged in the 
following teacher practices: (1) they made substantial use of the standards to structure their 
curriculum, (2) they made substantial changes in their teaching methods as a result of using the 
standards, and (3) they made substantial changes in their teaching methods as a result of using 
the CPA. Table 30 indicates that standards and the CPA had some impact on changes in teacher 
practices in both LCR and HCR schools. Teachers in HCR schools attributed more changes in 
their teaching practices to the CPA and students’ efforts for the CPA than did teachers in LCR 
schools. 
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Table 30: Percent of teachers who altered curriculum and teaching methods 



LCR HCR 

(N=40) (N=16) 

Some change None Some change None 



I used the standards to 

structure my curriculum. 88 

Using the standards led me 
to alter or rethink my teaching 
methods. 92 

Using the CPA as a means of 
organizing student work led 
me to alter or rethink my 
teaching methods 78 

Seeing student efforts for the 
CPA led me to alter or rethink 
my teaching methods. 



12 87 - 13 



8 100 0 



22 94 6 



29 87 13 



The student survey elicited comparative responses about teacher practices. When asked if 
teachers who use the CPA teach differently than non-CPA teachers, 46 percent of LCR students 
agreed or strongly agreed. At HCR schools, 68 percent agreed or strongly agreed and the percent 
of students that strongly agreed was triple that of the LCR schools. When asked if CPA teachers 
gave different assignments than non-CPA teachers, 52 percent of LCR students agreed/strongly 
agreed and 64 percent of HCR students agreed/strongly agreed. This corroborates teacher reports 
that the CPA influenced teaching practices, particularly related to use of standards. 



Table 31: Percent of students who perceive differences in teacher practices 



LCR HCR 

(N=626) ■ (N=436) 

SA A SA A 



Teachers who use the CPA teach 
differently than teachers who do 

not use the CPA 8 38 24 44 

Teachers who use the CPA ask me 
to do different kinds of assignments 

than teachers who do not use the CPA. 10 42 14 50 
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During interviews teachers were asked whether the CPA had an impact on their teaching. 
Positive comments fell into four categories: student achievement and growth, teacher personal 
growth, CPA as a vehicle for standards-based instruction, and CPA as an assessment tool. 
Among the comments were the following: 

(1) student achievement and growth: 

• It was good for us to see that it does work, that students do catch on and take pride in 
what they do. 

• Students were proud of what they had done. 

• Students developed greater organizational skills. 

• The CPA gave students a broader view of their education. 

• The CPA greatly impacted their career understanding and enhanced their use of 
technology. 

(2) teacher personal growth: 

• It allows us to organize. It will turn out to be a good integrating document. 

• It enabled us to do new and innovative things. 

• At first, it was more things to do, but I had never done a reflective piece where kids 
thought about what they did. That affected how I taught. 

• For some teachers it was a wonderful project, which gave us a chance to show all that 
can be achieved, tremendously interdisciplinary. 

(3) CPA as a vehicle for standards-based instruction: 

• When assigning work, we now look at standards and, therefore, we know why we are 
assigning a particular piece of work. 

(4) CPA as an assessment tool: 

• It helped me evaluate what I was doing in the classroom. 

• One can measure student achievement of standards without looking at the minutiae. 

• It provided a way for students to do self-evaluation. 

Some teachers felt the impact was minimal because they were already familiar with standards- 
based instruction and assessment. 

• Our teachers already understood standards, portfolio assessment; we were already 
doing integration, authentic assessment, project-based learning. 
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Some of the negative comments focused on additional time and effort needed to integrate the 
CPA into existing curriculum. Other comments included the following: 

• It was difficult to make it an integral part of the curriculum. It became a distraction to 
the curriculum. 

• It was an intrusion on the curriculum. 

Interviews with principals and lead teachers produced results that were consistent with teacher 
interviews. Their comments included the following: 

• Teachers experienced growth. 

• Not all teachers bought in - yet. We have work to do. 

• The impact came from the work summaries and the work samples required of every 
student. Our students needed a lot of one-on-one. The CPA put more work on 
teachers, but they put students’ interests first. 

• For teachers who bought in, it was a positive effect; for the ones who did not, it was a 
negative effect. 

• The process of corning to consensus as a staff on the outcomes was a positive 
experience. Teachers in different subject areas felt connected through the portfolio. 

• Teachers saw that students did much better work than they had expected - incredible 
work. 

• It made teachers see the relationship between standards and classroom performance. 

b. Use of scoring data in the classroom and the school . The teacher survey asked respondents to 
indicate whether they would like to receive CPA scoring data on individual students. In LCR 
schools, 75 percent of the respondents indicated they would like to receive the data; in HCR 
schools, 94 percent of respondents indicated they would like to receive the data. This seems to 
indicate more of a commitment in HCR schools to use the portfolio as an integral part of 
classroom work and grading structure than in LCR schools. 

The teacher survey also asked respondents to indicate the ways in which student performance on 
the CPA was incorporated into student grades or other relatively high stakes outcomes. More 
than 60 percent of teachers in LCR and HCR schools reported incorporating student performance 
on the CPA into the semester grade in their classes. More than 30 percent of teachers in LCR 
and HCR schools reported that student performance on the CPA was a requirement of either 
graduation or academy certification. In interviews with teachers, some respondents added to the 
survey options by indicating they reported CPA results to their students. Interviews with 
principals/lead teachers indicated that in some schools, scoring data were reported to students 
and student advisors. It also was used in some schools for program evaluation and course 
placement. 
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Table 32: Percent of teachers selecting options for incorporating student performance on 





LCR 


HCR 




(N=42) 


(N=17) 


A percentage of the semester grade in my class(es) 


62 


65 


A percentage of the semester grade in other class(es) 


14 . . 


24 


Graduation requirement 


26 


18 


Academy certification 


5 


12 


None of the above 


7 


6 


Other/don’t know 


31 


6 



In summary, the CPA had impact on the great majority of teachers that used it and that impact 
was identified by teachers and students. The extent of that impact is not known. Additionally, 
HCR schools almost always reported greater impact on teaching practice than LCR schools. This 
is consistent with the belief that HCR schools embraced the CPA more deeply than LCR schools. 

What is the Impact of the CPA on Teacher Understanding of Standards- 
Based Teaching, Assessment, and Portfolios? 

Data pertaining to the question come from interviews with teachers and principals/lead teachers. 
Nine of the 16 teachers interviewed reported the CPA had no impact on their understanding of 
standards-based teaching, assessment, and portfolios because they came into the program with a 
high level of understanding in each of the areas. The remaining seven teachers reported they had 
experienced personal growth and understanding and specifically mentioned a deeper 
understanding of assessment and the relationship between standards and performance. All of the 
principals/lead teachers interviewed felt the teachers in their schools had a greater understanding 
of standards-based teaching, assessment, or portfolios. Comments from these interviews 
included the following: 

• The CPA made teachers see the relationship between standards and classroom 
performance. 

• For some teachers it was the first experience with a portfolio. Tying the standards to 
the portfolio was very helpful. 

• It was a key to teachers’ growth and understanding. 

• It certainly helped for better understanding of assessment and portfolio. We worked 
with standards before the project, but it improved understanding on all three. 

What is the Impact of the CPA on Whole-school change. Particularly in 
Schools with Traditional Structures? 

Data come from teacher surveys and interviews with teachers and principals/lead teachers. The 
following issues were examined: (a) portfolios as a valuable support for implementing 
standards-based curriculum and assessment school-wide, (b) changes in the school/program 
structure, procedures, or curriculum, (c) the school’s plans for next year, (d) overall impact, and 
(e) the relationship of existing school structure to whole-school change. 
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a. Portfolios as a valuable support for implementing standards-based curriculum and assessment 
school-wide. Teacher surveys indicate there is general agreement in LCR and HCR schools that 
the CPA is valuable for implementing standards-based curriculum and assessment school-wide. 
Teachers in HCR schools, however, appear to feel much more strongly about the CPA as a 
valuable support for implementing standards-based curriculum and assessment school-wide 
(HCR 47 percent strongly agree, LCR 15 percent strongly agree). 



Table 33: Percent of teachers who strongly agree or agree on the value of the CPA for 
implementing standards-based curriculum and assessment school-wide. 

LCR HCR 

(N=39) (N=15) 

Strongly Agree 15 47 

Agree ^ 77 £7 



b. Changes in the school/program structure, procedures, or curriculum. Teacher surveys indicate 
that in the LCR schools, 32 percent of teachers report substantial change in their schools as a 
result of the CPA compared to six percent of teachers in HCR schools. The enabling structures 
discussed earlier in these findings provides an explanation of why HCR teachers report fewer 
changes. The enabling structures, already established in HCR schools, provide many of the key 
conditions required for successful implementation of the CPA. LCR schools, in contrast, had to 
make alterations in their structures, procedures, and/or curriculum to accommodate the CPA’s 
implementation. More than 85 percent of teachers in both LCR and HCR schools report that 
some change took place. Additionally, WestEd staff indicated they believe that many lead 
teachers and principals at LCR schools learned about their schools’ weaknesses relative to efforts 
that require whole-school change and implementation of innovations. WestEd staff also 
indicated that many teachers at LCR schools saw the portfolio as an add-on activity and that 
supportive LCR teachers thought they could implement the portfolio in their classrooms but that 
teacher support and school structures prevented comprehensive implementation across their 
schools. 



Table 34: Percent of teachers who report changes in school/program structure, procedures, 
or curriculum based on CPA related student work 





LCR 


HCR 




(N=39) 


(N=15) 


Substantially 


32 


6 


Slightly 


54 


81 


Not at all 


14 


13 



c. The school’s plans for next year. Teacher surveys asked respondents whether they intended to 
continue using the CPA next year. In LCR schools, 98 percent of the teachers indicated they 
planned to continue using the CPA next year; in HCR schools, 100 percent of the teachers 
indicated they planned to continue. 
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Interviews with principals/lead teachers indicated that 100 percent of the interviewees planned to 
continue, and to expand the CPA. Expansions included: add more students from the same grade 
in which the program was implemented this year, extend the program to other grade levels, 
extend the program to other academies in the school, and include more teachers in the program. 
One principal indicated a new course would be added to orient students to the CPA and another 
planned to include the CPA in an already existing orientation course for incoming students. 

Actual implementation data about LCR schools in the 1999-2000 school year appears to 
contradict the survey and interview data from LCR schools. No LCR schools implemented the 
portfolio on a school-wide basis in 1999-2000, nor did they implement the CPA at levels equal to 
1998-1999. Why LCR teachers and administrators reported that they would implement in 1999- 
2000 and did not do so is unclear. 

d. Overall impact. When asked directly to indicate whether the CPA had an impact on whole- 
school change, interviewed teachers indicated that their schools had become more standards- 
based, that the CPA was an important assessment tool, and that the CPA helped their schools in 
the state accreditation process. They pointed to plans to expand the program and to improve it. 
Interviews with principals/lead teachers indicated that whole-school change had not yet occurred 
in three schools, but anticipated that it would. In the five schools where principals/lead teachers 
reported whole-school change, interviewees pointed to plans for higher stakes, more teacher 
discussions with colleagues, better communication with incoming students, and student 
awareness of standards. 

One teacher summed up the impact of the CPA on her school as follows: 

It standardized a lot of what we do here so that classes were doing the same thing 
[focusing on the learning outcomes] and assessment of the learning outcomes applicable 
to our content areas. It was important that we all used a standard assessment. It brought 
the school together more. 

e. The relationship of existing school structure to whole-school change. Assessing the CPA’s 
impact on whole-school change has to be considered in the context of the schools’ existing 
structures and the implementation goals and strategies employed in the schools. These varied 
across the eight participating schools and included three options: (1) whole school, (2) academy 
or team, and (3) all students in a grade level (without a shared cohort of students). 

Three schools in the sample implemented the CPA on a whole-school basis. All of these schools 
had a small student body. One of the schools was newly created with a small staff recruited 
specifically because of their experience and expertise in carrying out the mission of the school. 
Some of the teachers in this school reported they had previous experience with portfolios, 
standards-based teaching and assessments, and integrated curriculum. The CPA proved to be an 
excellent vehicle for carrying out the school’s mission, and the entire staff worked on the 
tailoring process. This school produced a substantial number of completed portfolios. However, 
very little whole-school change could be reported here, since they began with much of the 
structures associated with whole-school change already in place. 
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The other two schools, both LCR, that implemented on a whole-school basis, were two of four 
high schools in a district, all of which had been committed to the CPA at the district level. 
Interview and survey data indicated enthusiasm for the CPA. However, two-thirds of the 
respondents from the two schools reported they needed more experience with the standards and 
almost half needed more experience with the CPA. Almost 90 percent of teachers across the two 
schools reported the school had changed its structure, procedures, or curriculum based on student 
work related to the CPA portfolio. An interview with the principal (one person js the principal of 
both schools), indicated that teachers were trying very hard to do the additional work imposed by 
the implementation of the CPA. These two schools did not produce a substantial number of 
completed portfolios, but the principal felt they would do so in the second year of the program. 
However, these two schools did not commit the resources needed to bring about whole-school 
change sufficient to implement the CPA in 1999-2000. 

In three of the four schools where the CPA was implemented on a team basis, many of the key 
preconditions were already in place at the beginning of the program: integrated curriculum, team 
teaching, and a shared group of students. Teachers in these three HCR schools reported only 
slight changes in teacher practices. 

Implementing the CPA by involving all students in a grade level without shared groups of 
students appears to be the most difficult of the three options. Two of the schools in the sample 
attempted to do so. The principals of both schools reported mixed success. Generally, for 
teachers who bought in to the CPA, there was a positive effect; for the others, the effect was 
negative. In both of the schools, the CPA was implemented across the entire curriculum for a 
particular grade. Across the two schools, one-fourth of the teachers responding reported the 
school had substantially changed its' structure, procedures, or curriculum based on the CPA, and 
half of the teachers reported slight changes. Three-quarters of responding teachers from the two 
schools felt they needed more school-wide support to successfully implement the CPA in their 
classrooms. 

From the teachers’ standpoint, there are certain logistical problems with implementing the CPA 
across a single grade level. A major problem appears to be not having significant numbers of 
students in common with other teachers using the CPA. A related concern is that a teacher in a 
particular academic content area may have classes of students from different grade levels. Also, 
there are issues pertaining to the storage and management of large numbers of student portfolios. 
Finally, there are issues pertaining to professional development. Both of the two schools have 
teaching staffs of 20-25 people. Teacher interviews indicated that participation in the tailoring 
process generates ownership; for those persons who did not participate, there is a feeling that the 
program “belongs” to the teachers who tailored the CPA. 

Neither of the two schools produced a substantial number of completed portfolios; however, one 
produced enough to conduct benchmarking scoring to identify exemplars of student work- at each 
score point on the rubric. 

In summary, there was general agreement in LCR and HCR schools that the CPA is a valuable 
tool for implementing standards-based curriculum and assessment school-wide. However, in 
terms of actually causing, versus supporting, substantial change in structures, procedures, or 
curriculum, teachers were less conclusive. It appears that in LCR schools, teachers that bought in 
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to the CPA did make changes. LCR teachers that did not buy in, and they appeared to be the 
majority at LCR schools, did not make changes. At HCR schools, deep structural and classroom 
changes were less likely as they already had enabling structures in place. On the whole, LCR 
schools would have had to make alterations school-wide in their structures, procedures, and/or 
curriculum to engender successful CPA implementation. If LCR schools had been able to change 
rapidly to create the enabling whole-school structures of the HCR schools (team teaching, shared 
students, teacher planning time, integrated curriculum, etc.), WestEd staff believe they would 
have had a good chance of having much greater completion rates. 
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CONCLUSION 



The 1998-1999 CPA effort generated data about the tailored CPA’s potential use as a scalable, 
reliable assessment that supports systemic reform and improves student performance. Major 
findings from the 1998-1999 project confirmed findings from the 1996-1997 cohort and shed 
light on the new issue of tailoring. Like the whole of this report, the conclusiort is organized 
around the, four research areas of tailoring, implementation, student performance and assessment, 
and whole-school change. 

Tailoring 

1. The tailoring process enabled schools to efficiently customize the CPA to their own 
needs, thus increasing ownership or “buy-in.” The opportunity to customize the CPA to 
local needs and standards made a substantial contribution to teacher buy-in, especially for 
those teachers who were participants in the tailoring process. 

2. The tailoring process appears to have ongoing substantive value, in addition to buy-in. 

Three of the four HCR schools revised their portfolio designs at the end of their first year. 
Similarly, three of the four 1999-2000 schools will revise their portfolio this summer. 

3. Tailoring allowed schools to use the CPA to meet regulatory, as well as curriculum and 
assessment needs thereby increasing the tailored CPA’s value. California CPA schools 
that had recently been through or were about to go through the state’s accreditation process 
reported that the state saw great value in the portfolio due to its emphasis on student work 
targeted at standards. 

4. The tailoring process did not place undue burden on schools. While the tailoring process 
did take two days of staff time, the fact that tailoring generally occurred in summer, along 
with the fact that teachers found the process to be useful in understanding the portfolio, 
schools generally found the time to be well spent. 

5. Tailoring is integral to long-term scalability. Tailoring allows schools to fully understand 
the portfolio, particularly how the standards, entries, and rubric are interdependent and how 
they link to curriculum. Once schools have this understanding, the portfolio becomes their 
own, as opposed to a model controlled by an external group. Schools can then modify their 
tailored portfolio in accordance with evolving content standards and performance levels. 



Implementation 

1 . The tailored CPA can be implemented successfully in schools possessing certain 
structural characteristics generally associated with schools that are engaged in whole- 
school reform. These characteristics are: 

• recognition of the benefit of standards-based teaching and assessment for students 

• strong teacher buy-in for the CPA as the right vehicle to implement standards-based 
teaching and assessment 

• strong school leadership 

• commitment to reform 

• whole-school focus by teachers (as opposed to strong focus on individual departments) 

• integrated curriculum 

• interdisciplinary/team teaching 

• block scheduling 

• a shared cohort of students 

2. In CPA pilot tests over four years, these conditions were most prevalent in small 
schools, or smaller units within schools (e.g., academies, career paths). This suggests 
that the CPA is most sustainable and scalable in smaller learning environments. All 

four of the 1988-1999 HCR schools are small and are committed to using the CPA in the 
2000-2001 school year, after WestEd support has ended. Furthermore, of the cohort of four 
schools that started the CPA in the 1999-2000 school year, and that were chosen with more 
emphasis on the structural characteristics listed above than the schools chosen for 1998-1999, 
the three small ones were most successful. (All four will continue to implement in the 2000- 
2001 school year.) 

3. Like other substantial assessment and curriculum tools that require schools to be well 
down the reform path, the CPA is not likely to be scalable across a majority of high 
schools given the current reform status of most high schools. Most high schools do not 
possess a preponderance of the conditions listed above, particularly across the entire school. 
Furthermore, the CPA was notably successful in smaller environments such as academies and 
has been less successful in larger schools. This is not surprising given the structural 
characteristics listed above — large schools are less likely than small schools to possess them. 
Schools not possessing a preponderance of the conditions for success would do well to 
engender a whole-school (versus departmental) focus, examine why the conditions for 
success do not exist, and proceed to create the conditions. 

4. As in 1996-1997, survey and interview data from teachers and students were positive 
about the value of the CPA. Teachers across disciplines saw value in the CPA. Perhaps as 
importantly, the schools possessing the conditions for success appear to have institutionalized 
the CPA to varying degrees. This is important as the tailored CPA requires a major, ongoing 
effort by teachers and students. Simply put, in schools that are relatively far down the reform 
path, the only challenging, non-mandated reforms that last are the ones that teachers believe 
positively impact students. 
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5. Scoring large numbers of portfolios challenges schools’ ability to provide scorers and 
could impact scalability. The 1998-1999 schools scored under 200 portfolios each and had 
some difficulty in recruiting scorers. The resources necessary to score large numbers of, 
portfolios in single scoring sessions could make meaningful scoring extremely difficult for 
large schools. Alternative scoring models, such as classroom-based scoring, need to be 
explored. 

Student Performance and Assessment 

1 . Reliable scoring of the tailored CPA requires a commitment to conducting scoring with 
substantial training. Scoring in the 1998-1999 pilot test occurred in conditions not 
conducive to high reliability. Not surprisingly, reliability achieved was not sufficient for 
high-stakes decisions such as graduation requirements or academy certifications based on 
portfolio quality, as opposed to just portfolio completion. In the 1999-2000 effort, where 
WestEd staff focused on increasing training time, reliability was higher than in 1998-1999 (a 
range of 64 percent to 79 percent versus a range of 58 percent to 71 percent). WestEd staff 
more aggressively communicated the need for reliability to participating schools in 1999- 
2000. Also in 1999-2000, WestEd staff targeted training at scoring dimensions where it was 
low in 1998-1999 (e.g.. Technology, Analysis). 

2. Student performance on the tailored CPA was acceptable to implementing teachers. 

Similar to the 1996-1997 effort, 68 percent of 1998-1999 students who completed portfolios 
“passed,” or received a Proficient or Advanced score, and 32 percent “failed,” or received a 
Basic. 1999-2000 student performance was similar to 1998-1999, with 66 percent of students 
receiving a Proficient or Advanced score, and 34 percent receiving Basic. These ratios of 
Proficient/ Advanced to Basic have not deterred HCR schools from continuing to implement 
the CPA. 

Whole-school change 

1 . A strong majority of 1998-1999 project teachers believe the tailored CPA is a valuable 
teaching tool that promotes change in classroom practice and can support school-wide 
reform. It should be noted that in HCR schools many key structural conditions for 
successful implementation were already well established at the outset of CPA 
implementation. There was not the impetus, nor the need, for substantial structural change at 
these schools. In these schools, the CPA acted as a curricular tool and focus for what 
students should know and be able to do at the school-wide level, and as an opportunity to 
implement high quality standards-based curriculum and assessment. For schools not 
possessing the key conditions, the CPA acted as a litmus test for implementation of any 
school-wide reform. After attempting to implement the tailored CPA, leaders at LCR schools 
more clearly understood what structural and attitudinal changes needed to occur. 

2. The tailored CPA allowed and supported “ready” schools to implement their visions 
related to having a school-wide student performance focus and to standards-based 
teaching and learning. One principal of an HCR school stated it most clearly, “The 
portfolio is the flag in the ground for our vision. It gives our vision definition and helps to 
make it a classroom reality.” 




58 



ro 



The capacity of any school not possessing a preponderance of the CPA’s conditions for success 
to implement any type of standards-based reform on a school-wide basis is highly questionable. 
Given that, the fact that the CPA did not fare well in LCR schools should not be surprising or 
disconcerting. Moreover, that the CPA is flourishing in schools with faculties acutely attuned to 
innovation and in schools possessing the conditions for success is strong testimony to its value. 

In our national context where precious few high schools are implementing standards-based 
reform in more than a few departments, the CPA is an effective tool to help scale standards-based 
teaching and learning across whole schools and across ready subsets of schools. 
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Appendix A 

Career Preparation Standards 



Career Preparation Standards (CPS) 



1. Personal Skills. Students wUl understand how personal skill development 
affects their employabUity. They will exhibit positive attitudes, self confidence, 
honesty, perseverance, self-discipline, and personal hygiene. They will manage 
time and balance priorities as well as demonstrate arcapacity for lifelong learning. 

2. Interpersonal Skills, students will understand key concepts in group 
dynamics, conflict resolution, and negotiation. They will work cooperatively, share 
responsibilities, accept supervision, and assume leadership roles. They will 
demonstrate cooperative working relationships across gender and cultural groups. 

3. Thinking and Problem Solving Skills. Students wUi exhibit critical 
and creative thinking skills, logical reasoning, and problem solving. They will 
apply numerical estimation, measurement, and calculation, as appropriate. They 
will recognize problem situations; identify, locate, and organize needed information 
or data; and propose, evaluate, and select from alternative solutions. 

4. Communication Skills. Students wUl understand principles of effective 
communication. They will communicate both orally and in writing. They will 
listen attentively and follow instructions, requesting clarification or additional 
information as needed. 

5. Occupational Safety. Students will understand occupational safety issues 
including the avoidance of physical hazards in the work enviromnent. They will 
operate equipment safely so as not to endanger themselves or others. They will 
demonstrate proper handling of hazardous materials. 

6. Employment Literacy. Students will understand career paths and strategies 
for obtaining employment within their chosen fields. They will assume 
responsibility for professional growth. They wUl understand and promote the role 
of their field within a productive society, including the purpose of professional 
organizations. 

7. Technology Literacy. Students will understand and adapt to changing 
technology by identifying, learning, and applying new skills to improve job 
performance. They will effectively employ technologies relevant to their fields. 
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Appendix B 

CPA Rubric 



Career Preparation Assessment (CPA) Rating Guide 
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Examples of “Tailored” Rubrics 
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Appendix D 

Examples of “Tailored” Standards 



Alvarez School to 
Career Skills 

Overview The Alvarez School to Career Skills (ASCS) state the knowledge and 

skills necessary to successfully develop your career. You will need 
to strengthen these skills to succeed in advanced education, training, 
and on the job. Because achieving and demonstrating the ASCS is 
critical to completing your ASCA portfolio, you need to know the 
ASCS well. 

1. Personal Skills 

Good personal skills are very important in school and in work. 
Iristructors and employers expect you to be responsible, self-disciplined, 
professional, and able to manage your priorities well. In addition, they 
expect you to be free from substance abuse and to dress in a professional 
manner. Honesty and personal integrity are also expected on any job. 

Examples of personal skills include: 

• ethical behavior — do what you know is right even when it is not 
the easiest choice 

• perseverance — complete a task even when it is difficult 

• time management — meet deadlines and arrive to appointments on 
time 
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2. Interpersonal Skills 

Today's classrooms and workplaces require that you be able to work as 
part of a team. You need to be able to work cooperatively with people of 
different ages and cultural backgrounds. When working with others it 
is important to understand how to resolve conflicts, negotiate, share 
responsibilities, accept supervision, and assume leadership roles. 

Examples of interpersonal skills include: 

• leadership — assign work and inspire co-workers to complete tasks 

• following — accept a decision and help meet a goal 

• teamwork — use group planning and goal-setting techniques to 
solve problems 

3 . Analysis and Problem-Solving Skills 

There are many ways to solve a problem. Good analysis and problem- 
solving skills will help you contribute solutions on the job. Employers 
and colleges will expect you to be able to think analytically and 
creatively, use logical reasoning, and interpret information. You may 
need to apply skills such as numerical estimation, measurement, 
calculation, and evaluation to solve problems. 

Examples of analysis and problem-solving skills include: 

• problem definition — recognize a problem and identify the 
relevant facts 

• analysis — gather and evaluate information 

• decision making — identify and compare possible solutions, and 
choose the most appropriate 
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4 . Written Communication Skiiis 



Strong written communication skills are very important for school, for 
most jobs, and will help you in your job search. You must be able to 
read and write effectively. Effective writing means that you can: 
organize information, clearly present your own ideas, write with your 
audience in mind, and use correct grammar and spelling. 

Examples of written communication skills include: 

• composing — write effectively, using good sentence structure and 
vocabulary 

• editing — revising what you have written to improve clarity 
and/or organization of ideas 

• presentation — know who the reader is and communicate written 
ideas effectively to your audience 



5 . Orai Communication Skiiis 

Oral communication skills are very useful in school and are of particular 
importance when entering and advancing in a career. In an interview 
and on the job, you will need to present information and ideas 
effectively. 

Examples of oral communication skills include: 

• knowing your audience — anticipate the previous knowledge of 
your audience and provide the information they will need 

• presenting information — speak clearly to convey your ideas to 
others; introduce information in a logical order 

• using audio/visual aids — provide graphics or visuals which 
enhance your message 
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6. Employment Literacy 

Due to the changing demands of the job market, you need to have a 
good understanding of career paths and how to find jobs in your chosen 
field. It is important for you to develop a career plan that describes your 
goals and plan of action. In order to develop a career plan, you need 
information about yourself, requirements for education or training, and 
the job market. 

Examples of employment literacy include: 

• knowledge of your skills — be aware of personal strengths, 
weaknesses, interests and abilities, and how they relate to specific 
career areas 

• awareness of the job market — research current trends in the job 
market, such as jobs that are in high or low demand 

• knowledge of career paths — investigate possible careers and 
entry-level positions with potential for growth 



7 . Technology Literacy 

Many jobs require computer skills, technical knowledge, and the ability 
to learn and adapt to new technologies. Employers expect you to be able 
to use technology to complete tasks and to improve your job 
performance. 

Examples of technology literacy include: 

• computer skills — select and use appropriate hardware and 
software 

• audio and video skills — use audio and video equipment to 
document events effectively 

• technical knowledge — solve a problem based on your knowledge 
of technology 
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Professional Skills 
Standards 

0v6rvi6W The Professional Skills Standards (PSS) state the knowledge and 

skills necessary for success beyond high school. You will need to 
strengthen these skills to succeed in advanced education, training, 
and on the job. Because achieving and demonstrating the PSS is 
critical to completing your PSA portfolio, you need to know the PSS 
well. The seven skills cover: 

1. Personal Skills 

2. Interpersonal Skills 

3. Thinking and Problem-Solving Skills 

4. Communication Skills 

5. Project Management 

6. Professional Awareness 

7. Technology Skills 

Each standard is described in detail below. 



1. Personal Skills 

Students will understand how personal skill development affects 
their employability. They will exhibit positive attitudes, self 
confidence, integrity, perseverance, self-discipline, responsibility, 
and craftsmanship. They will manage time and balance priorities as 
well as demonstrate a capacity for growth, development and self- 
reflection. 



2. Interpersonal Skills 

Students will understand key concepts in group dynamics, conflict 
resolution, and negotiation. They will work cooperatively, share 
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responsibilities, accept supervision, and assume leadership roles. 
They will demonstrate cooperative working relationships across 
gender and cultural groups. 

3. Thinking and Probiem-Soiving Skiiis 

Students will exhibit critical and creative thinking skills, logical 
reasoning, and problem-solving. They will understand systems. 
They will apply numerical estimation, measurement, and 
calculation, as appropriate. They will recognize problem situations; 
identify, locate, and organize needed information or data; and 
propose, evaluate, and select from alternative solutions. 



4. Communication Skiiis 

Students will understand principles of effective communication. 
They will communicate both orally and in writing to a range of 
audiences in a variety of ways. They will listen actively and follow 
instructions, requesting clarification or additional information as 
needed. 



5 . Project Management Skiiis 

Students will understand and use project planning skills. They will 
set goals and develop strategies to meet those goals in a timely 
manner. They will acquire and use information and resources to 
implement their strategies. They will evaluate the effectiveness of 
their approach. 

6. Professionai Awareness 

Students will understand the necessary skills, educational 
requirements, and professional opportunities in chosen fields. 
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7 . Technology Skills 

Students will know when it is appropriate to use technology, what 
technology to use, and how to use it to perform tasks and enhance 
their work. 
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Pride Standards 

Overview The Pride Standards state the knowledge and skills necessary to 

successfully develop your career. You wiU need to strengthen these 
to succeed in advanced education, in training, and on the job. 
Because achieving and demonstrating the Pride Standards is critical 
to completing your P^ you need to know the standards well. The 
seven standards are: 

1. Personal Standard 

Students will exhibit how personal development affects their 
employabihty, including positive attitudes, self-confidence, honesty, 
perseverance, self-discipHne, social responsibiUty, and personal 
hygiene. They will manage time and resources, balance priorities, 
and demonstrate a capacity for self-reflection and Hfelong learning. 

2. Interpersonal Standard 

Students will participate in group dynamics, conflict resolution, and 
negotiation. They will work cooperatively, share responsibiUties, 
accept supervision, and assume leadership roles. They wiU 
demonstrate cooperative working relationships across gender and 
cultural groups. 



3. Thinking and Problem-Solving Standard 

Students will exhibit critical and creative thinking, logical reasoning, 
and problem-solving. They will apply numerical estimation, 
measurement, and calculation, as appropriate. They will recognize 
problem situations; identify, locate, and organize relevant 
information or data; and propose, evaluate, and select from 
alternative solutions. 
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4. Communication Standard 



Students will demonstrate the use of the principles of effective 
conununication. They will communicate both orally and in writing. 
They will listen actively and follow instructions, requesting 
clarification or additional information as needed. 



Students will select and design career paths and strategies for 
obtaining employment within their chosen fields. Students will 
demonstrate occupational safety including the avoidance of physical 
hazards in the work environment. They will assume responsibility 
for professional growth. They will promote the role of their field 
within a productive society, and understand the purpose of 
professional organizations. 



Students will demonstrate the ability to select the appropriate 
technologies and effectively employ those technologies to their 
chosen fields. 



Students will demonstrate the ability to apply knowledge and skills 
from across disciplines to develop strategies for, and solutions to, 
real world situations. 



5 . Empioyment Literacy 



6. Technoiogy Literacy 



7. Making Connections 
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NTHS Learning 
Outcomes 



Ov©rvi©W The NTHS Learning Outcomes state the knowledge and skills 

necessary for success beyond high school. You will need to 
strengthen these skills to succeed in advanced education, training, or 
the work place. Because achieving and demonstrating the Learning 
Outcomes is critical to completing your Graduation Portfolio, you 
need to know the Learning Outcomes well. The eight Learning 
Outcomes are: 

1. Technology Literacy 

2. Citizenship and Ethics 

3. Critical Thirdcing 

4. Career Preparation 

5. Collaboration 

6. Written Communication 

7. Oral Communication 

8. Curricular Literacy 

Each outcome is described in detail below. 



1 . Technology Literacy 

Many jobs require computer skills, technical knowledge, and the ability 
to learn and adapt to new technologies. Employers expect you to be able 
to use technology to complete tasks and to improve your job 
performance. 

Examples of technology literacy include: 

• typing skills — type 40 WPM with 90% accuracy 

• software skills — 80% accuracy in presentation software, word 
processing, spreadsheet, database, and animation modules 
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• multimedia skills — create a project that incorporates sound, 
animation, and interactivity 

• technology communication skills — create web pages and gather 
information from the internet and a variety of technological media 

2. Citizenship and Ethics 

Being successful does not only pertain to academic or financial 
achievement. The most successful individuals are those who also 
purposely contribute to their community and conscientiously do what 
they know is right, even when it is not the easiest choice. 

Examples of citizenship and ethics include: 

• student government — get involved in important school decisions 

• service learning/ volunteering — donate time and energy to 
support a cause that needs your help 

• peer tutoring — help a fellow student understand coursework you 
have mastered 

3. Critical Thinking 

There are many ways to solve a problem. Good thinking and problem- 
solving skills will help you contribute solutions on the job. Employers 
and colleges will expect you to be able to think analytically and 
creatively, use logical reasoning, and interpret information. You may 
need to apply skills such as numerical estimation, measurement, 
calculation, and evaluation to solve problems. 

Examples of thinking and problem-solving skills include: 

• problem definition — recognize a problem and identify the 
relevant facts 

• organization — identify and gather information needed to solve a 
problem 

• analysis — evaluate and select information, and consider possible 
solutions to a problem 
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4 . Career Preparation 

In order to prepare for a career, you need to learn responsibility, self- 
discipline, and time management. Honesty and personal integrity are 
also expected on any job. In addition, it is important to have a good 
understanding of career paths and how to find jobs in your chosen field. 
You will develop a career plan that describes your goals and plan of 
action. In order to develop a career plan, you need to compile 
information about yourself, information about requirements for 
education or training, and information about the job market. 

Examples of career preparation include; 

• positive attitude and perseverance — willingly follow directions, 
take on tasks and responsibilities, complete a task even when it is 
difficult 

• flexibility and initiative — adapt to change, apply your knowledge 

• knowledge of your skills — be aware of personal strengths, 
weaknesses, interests and abilities, and how they relate to specific 
career areas 

• knowledge of career paths — investigate possible careers and 
entry-level positions with potential for growth 



5 . Collaboration 

Today's classrooms and workplaces require that you be able to work as 
part of a team. You need to be able to work cooperatively with people of 
different ages and cultural backgrounds. When working with others it 
is important to understand how to resolve conflicts, negotiate, share 
responsibilities, accept supervision, and assume leadership roles. 

Examples of collaboration include; 

• leadership — assign work and inspire co-workers to complete tasks 

• following — accept a decision and help meet a goal 

• teamwork — use group planning and goal-setting techniques to 
solve problems 

O . 
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6. Written Communication 



Strong written communication skills are important in school, in the job 
search, and finally in most jobs. You must be able to read and write 
effectively. Effective writing means that you can: organize information, 
clearly present your own ideas, write with your audience in mind, and 
use correct grammar and spelling. 

Examples of written communication skills include: 

• composing — write effectively, using good sentence structure and 
vocabulary 

• editing — revising what you have written to improve clarity 
and/or organization of ideas 

• presentation — know who the reader is and communicate written 
ideas effectively to your audience 

7. Orai Communication 

Oral communication skills are very useful in school and are of particular 
importance when entering and advancing in a career. In an interview 
and on the job, you will need to present information and ideas 
effectively. 

Examples of oral communication skills include: 

• knowing your audience — anticipate the previous knowledge of 
your audience and provide the information they will need 

• presenting information — speak clearly to convey your ideas to 
others; introduce information in a logical order 

• using audio/ visual aids — provide graphics or visuals which 
ertiiance your message 
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8 . Curricular Literacy 

In addition to the life skills outlined above, the State of California and 
New Technology High School have identified key knowledge that 
students should have when they graduate from high school. Your 
academic coursework was designed to provide the educational 
foundations that will foster informed perspectives and a level of cultural 
literacy sufficient for success beyond high school. Therefore, your 
Graduation Portfolio should include evidence of your curricular literacy 
in as many areas as possible. 

Examples of curricular literacy skills include: 

• earning a certificate of proficiency (COP) — demonstrate your 
knowledge by earning a COP in each of your academic courses 

• succeeding on standardized tests — earn a score of "school 
recognition" or better on Golden State Exams or a score of 3 or 
better on Advanced Placement Exams 
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Appendix E 

Data Collection Instruments 



STUDENT SURVEY, 1998-99 

Professional Skills Assessment Portfolio (PSAP) 

This is a survey about the Professional Skills Assessment Portfolio (PSAP) that you 
have been involved in this year. Because the Professional Skills Assessment Portfolio 
(PSAP) is new, we are collecting information to help improve it. Please answer the 
survey questions as honestly and completely as you can. Your feedback is critical to 
us! Thank you for your time and thoughtfulness. 

1. Age: (circle one) 

13 14 15 16 17 18 19 

2. Grade: (circle one) 

9 10 11 12 

3. Gender: (circle one) 

Male Female 

4. Ethnicity: (circle one) 

a. American Indian 

b. Asian 

c. Pacific Islander 

d. Filipino 

e. Hispanic 

f. Black 

g. White 

h. Other (please describe): 

i. Decline to state 

5. Name of the class or classes where you worked on the Professional Skills 
Assessment Portfolio (PSAP). 

a. ■ 

b. ' 

c. 

d. 

e. 

6. Did you complete all the entries in the PSAP (Personal Statement, Resume, 
Application, Letter of Recommendation, to Work Samples, Writing Sample, 
Interpersonal Skills Evaluation, Oral Communication Skills Evaluation )? (circle 
one) 
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Yes 



No 



ABOUT THE PROFESSIONAL SKILLS STANDARDS (PSS) 

DIRECTIONS: Please circle the response that most closely corresponds to how much 
you agree with each of the statements in the survey. 

The Professional Skills Standards (Personal, Interpersonal, Thinking and Problem- 
Solving, Communication, Project Management, Professional Awareness! 

Technology) are the skills the PSAP measures. 

7. I understood the Professional Skills Standards (PSS). 

Strongly Strongly 

Agree Agree Disagree Disagree 

8. My teachers spent enough time explaining the PSS. 

Strongly Strongly 

Agree Agree Disagree Disagree 



9. Employers will hire people who have the PSS. 

Strongly Strongly 

Agree Agree Disagree Disagree 

10. Colleges will admit students who have the PSS. 

Strongly Strongly 

Agree Agree Disagree Disagree 

1 1 . Having the PSS will help me in the future. 

Strongly Strongly 

Agree Agree Disagree Disagree 

12. My teachers think it is important for me to learn the PSS. 

Strongly Strongly 

Agree Agree Disagree Disagree 



Do Not 
Know 



13. The PSAP is a good way to learn and master the PSS. 

Strongly Strongly 

Agree Agree Disagree Disagree 
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14. I know when I have mastered the PSS. 



Strongly Strongly 

Agree Agree Disagree Disagree 



ABOUT COMPLETING THE PSAP 

15. I understood how to do the entries that make up the PSAP (Personal Statement, 
Resume, Application, Letter of Recommendation, to Work Samples, Writing 
Sample, Interpersonal Skills Evaluation, Oral Communication Skills Evaluation). 

Strongly Strongly 

Agree Agree Disagree Disagree 



16. Overall, I received all the information, help, and support I needed to be successful 
in doing the PSAP. 

Strongly Strongly 

Agree Agree Disagree Disagree 



17. I received a copy of the PSAP “Guidelines For Students.” 

Yes No 

18. If you answered “Yes” to #17, were the PSAP “Guidelines For Students” useful? (If 
you answered “No” to #17, skip this question and go to #1 9.) 

Yes No 



19. Doing the PSAP will help me in the future. 



Strongly Strongly Do Not 

Agree Agree Disagree Disagree Know 

20. I recommend that other students do the pSAP. 



Strongly Strongly 

Agree Agree Disagree Disagree 
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ABOUT THE VALUE OF YOUR PSAP 



21 . Having the PSAP will help me get a job. 

Strongly Strongly Do Not 

Agree Agree Disagree Disagree Know 



22. Having the PSAP will help me get into college. 

Strongly Strongly Do Not 

Agree Agree Disagree Disagree Know 



23. 1 learn and master the PSS better in classes where I use the PSAP than in classes 
where I do not use the PSAP. 



Strongly Strongly 

Agree Agree Disagree Disagree 



24. Working on the PSAP entries helps to improve my academic work. 

Strongly Strongly 

Agree Agree Disagree Disagree 

25. Teachers who use the PSAP teach differently than teachers who do not use the 
PSAP. 



Strongly Strongly 

Agree Agree Disagree Disagree 

26. Teachers who use the PSAP ask me to do different kinds of assignments than 
teachers who do not use the PSAP. 

Strongly Strongly 

Agree Agree Disagree Disagree 

27. My teachers thought the PSAP was important. 

Strongly Strongly 

Agree Agree Disagree Disagree 

28. The PSAP is a good teaching and learning tool. 

Strongly Strongly 

Agree Agree Disagree Disagree 
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ABOUT IMPROVING THE PSAP 



29. If you were the teacher, how would you change the PSAP? (Circle as many as 
you wish.) 

a. I would make no changes. 

b. I would devote more time to explaining how to do the PSAP. 

c. I would allow more class time for students to work on the PSAP. 

d. I would provide more examples to guide the students’ work. 

e. Other. (Please explain) 



Please write on the back of this page if you need more room. 
THANK YOU VERY MUCH!! 
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TEACHER SURVEY, 1998-99 

Professional Skills Assessment (PSA) 

This is a survey about the Professional Skills Assessment (PSA). Because the PSA is 
new, we are collecting information to help guide its development. Your feedback in this 
survey is critical to the PSA. Please answer as honestly and completely as you can. 
You should be able to complete this survey in 15 minutes or less. When you have 
completed the survey, please give it to Bob Lenz. 

ABOUT YOUR TEACHING AND CLASSES 

1 . How long have you been teaching? 

0-1 years 2-3 years 4-9 years 10 or more years 



2. Have you ever implemented a portfolio before the PSA? 

Yes No 

3. What subjects do you currently teach (circle all that apply)? 

Agriculture Construction/Engineering Health 

Business Economics Math 

Career Awareness/Dev. English Science 

Computer Science Foreign Language Social Studies/History 

Other (please describe): 

4. In which subject(s) did you use the PSA? 

1. 2. 3. 



5. In how many of your classes did you use the PSA? 

1 2 3 4 5 

6. In the classes where you used the PSA, what percentage of class time was 

devoted exclusively to the PSA? % 
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7. In your classes where students were supposed to work on the PSA, what 
percentage; 

completed M entries of the PSA 

only completed some entries of the PSA 

did 0 ^ complete any entries of the PSA, but attempted 

some entries 

did n^ attempt any entries of the PSA 

(Percentages must total 100.) 

8. Did your students receive PSA Student Guidelines? yes no 

9. To what extent did the Guidelines help students do the PSA entries? 

Very helpful Somewhat helpful Slightly helpful Not helpful 

10. In what month did you introduce the Professional Skills Standards (PSS) to your 
students? 

Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May 

11. In what month did your students begin to create their first PSA entry? 

Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May 
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12. Student performance on the PSA was counted toward the following (circle all 
that apply): 

a. a percentage of the semester grade in my class(es) 

(estimate the percentage ) 

b. a percentage of the semester grade in other classes 

(estimate the percentage ) 

c. graduation requirement 

d. academy certification 

e. none of the above 

f. other (Please describe) ^ 



g. don’t know 

ABOUT THE PROFESSIONAL SKILLS STANDARDS (PSS) AND PSA OVERALL 

The following are a series of statements about the PSA. Please circle the response 
which best shows how much you agree or disagree with each statement. 

1 3. The school administration or teacher leadership communicated the purpose of 
the PSA clearly. 

Strongly agree Agree Disagree Strongly disagree 

14. Teachers and administrators in my school have a strong commitment to the 
PSA/PSS. 

Strongly agree Agree Disagree Strongly disagree 

1 5. The PSA is important for my students. 

Strongly agree Agree Disagree Strongly disagree 

16. My students were motivated and interested in participating in the PSA. 

Strongly agree Agree Disagree Strongly disagree 
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17. The Professional Skills Standards (PSS) represent skills valued by employers. 
Strongly agree Agree Disagree Strongly disagree 

1 8. The PSS represent skills valued by post-secondary educational institutions. 

Strongly agree Agree Disagree Strongly disagree - 

1 9. The PSA is a valuable assessment tool for me. 

Strongly agree Agree Disagree Strongly disagree 

20. The PSA assesses the PSS well. 

Strongly agree Agree Disagree Strongly disagree 

21 . Doing the PSA is a good way for students to learn or master the PSS. 

Strongly agree Agree Disagree Strongly disagree 

22. Using the PSA improves students’ academic work. 

Strongly agree Agree Disagree Strongly disagree 

23. PSAs are of interest to employers. 

Strongly agree Agree Disagree Strongly disagree 

24. PSAs are of interest to post-secondary educational institutions. 

Strongly agree Agree Disagree Strongly disagree 

25. Using PSAs is, or could be, a valuable support for implementing standards- 
based curriculum and assessment schoolwide. 

Strongly agree Agree Disagree Strongly disagree 

26. I would like to receive PSA scoring data on my students. 

Strongly agree Agree Disagree Strongly disagree 
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27. I intend to continue using the PSA next year. 

Strongly agree Agree Disagree Strongly disagree 

28. In this school, I am able to structure my curriculum as I choose. 

Strongly agree Agree Disagree Strongly disagree 

For statements 29 through 33, circle the response that indicates the extent to which the 
PSA led to changes in your school. 

29. I used the PSS to structure my curriculum. 

Very substantially Substantially Slightly Not at all 

30. Using the PSS led me to alter or rethink my teaching methods. 

Very substantially Substantially Slightly Not at all 

31 . Using the PSA as a means of organizing student work led me to alter or rethink 
my teaching methods. 

Very substantially Substantially Slightly Not at all 

32. Seeing student efforts for the PSA led me to alter or rethink my teaching 
methods. 

Very substantially Substantially Slightly Not at all 

33. Our school or program has changed its structure, procedures, or curriculum 
based on PSA related student work. 

Very substantially Substantially Slightly Not at all 
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34. What factors or resources contributed to the success of the PSA in your 
classroom? Please circle the 5 most important factors. 

a. Integrated curriculum 

b. Previous experience with standards-based assessment 

c. Previous experience with performance-based assessment 

d. Block scheduling 

e. Significant numbers of students in common with other teachers using the 
PSA 

f. Team teaching 

g. Career academies 

h. Professional development related to the PSA 

i. More class time for the PSA 

j. More experience with the PSS 

k. More experience with the PSA 

l. PSA Student Guidelines 

m. Students with more experience with portfolios 

n. More parent understanding of portfolios 
0 . More schoolwide support of the PSS 

p. More schoolwide support of the PSA 

q. Other (Please list): 
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35. What factors or resources that you did not have this past year would have made 
the PSA more successful in your classroom? Please circle the 5 most important 
factors. 

a. Integrated curriculum 

b. Previous experience with standards-based assessment 

c. Previous experience with performance-based assessment • 

d. Block scheduling 

e. Significant numbers of students in common with other teachers using the 
PSA 

f. Team teaching 

g. Career academies 

h. More professional development related to the PSA 

i. Different professional development related to the PSA than we received 

j. More class time for the PSA 

k. More experience with the PSS 

l. More experience with the PSA 

m. Different PSA Student Guidelines 

n. Students with more experience with portfolios 
0 . More parent understanding of portfolios 

p. More schoolwide support of the PSS 

q. More schoolwide support of the PSA 

r. Other (Please list): 



PROFESSIONAL DEVELOPMENT FOR THE PSA 

This section is designed to gather information about the professional development that 
you received for the PSA, now that you have worked with it. Please circle the response 
that best shows how much you agree or disagree with each of the following statements. 

After the PSA professional development workshops, I understood: 



36. the purposes of the PSA. 



Strongly agree 


Agree 


Disagree 


Strongly disagree 


37. the PSS. 








Strongly agree 


Agree 


Disagree 


Strongly disagree 
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Disagree Strongly disagree 



38. the entries in the PSA. 

Strongly agree Agree 

39. the nature of tasks that elicit work demonstrating the PSS. 

Strongly agree Agree Disagree Strongly disagree ‘ 

40. how to judge when student work demonstrates the PSS. 

Strongly agree Agree Disagree Strongly disagree 

41 . how to use the PSA rubric. 

Strongly agree Agree Disagree Strongly disagree 

42. how to design assignments which help students demonstrate the PSS. 

Strongly agree Agree Disagree Strongly disagree 

43. how to integrate the PSA into my curriculum plans. 

Strongly agree Agree Disagree Strongly disagree 

44. Overall, the workshops prepared me adequately to implement the PSA in my 
classes. 

Strongly agree Agree Disagree Strongly disagree 

45. I could have been better prepared to implement the PSA if more time had been 
devoted to professional development from WestEd. 

Strongly agree Agree Disagree Strongly disagree 

46. I could have been better prepared to implement the PSA if the professional 
development from WestEd had been of higher quality. 

Strongly agree Agree Disagree Strongly disagree 
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CREATED/TAILORED PORTFOLIO 



47. Were you aware that the PSA was created/tailored specifically for use in your 
school by teachers from your school working with WestEd staff? 

Yes No 

48. If yes, did you participate in this process? 

Yes No 

49. How many other teachers in your school who used the PSA in their classes do 
you think were aware that the program was created/tailored specifically for use in 
your school? 

Almost all Many Some Almost none 

50. The PSA measures standards already in use in my curriculum. 

Strongly agree Agree Disagree Strongly disagree 

51. Other comments: 
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SCORER SURVEY, 1998-99 



This is a survey about the scoring/benchmarking of the portfolios. We are collecting 
information to help in the improvement of the process. Your feedback in this survey is 
very important. Please answer as honestly and completely as you can. You should be 
able to complete this survey in 10 minutes or less. When you have completed the 
survey, please give it to the WestEd staff member leading your scoring/benchmarking 
event. 

1 . Name of School 

2. Please circle the category that describes you. 

a. Teacher who implemented the portfolio 

b. Teacher who did not implement the portfolio 

c. Site administrator 

d. District administrator 

e. Parent 

f. Employer 

g. Other (please specify: ) 

Please circle the response that best shows how much you agree or disagree with each 
of the following statements: 

3. Scoring/benchmarking the portfolios was a valuable experience. 

Strongly Strongly 

Agree Agree Disagree Disagree 

4. After scoring/benchmarking, I am better prepared to help my students produce 
Proficient portfolios. 

Strongly Strongly 

Agree Agree Disagree Disagree N/A 

5. Scoring/benchmarking students’ portfolios increased my knowledge of how well 
those students have mastered the portfolio’s standards. 

Strongly Strongly 

Agree Agree Disagree Disagree 

6. The time devoted to training for scoring/benchmarking the portfolios was 



adequate. 



Strongly 

Agree 



Agree 



Disagree 



Strongly 

Disagree 
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7. The content of the training I received to prepare me for scoring/benchmarking 
the portfolios was adequate. 

Strongly Strongly 

Agree Agree Disagree Disagree 

8. I feel confident that the school will be able to effectively conduct portfolio 
scoring/benchmarking next year without assistance/training from WestEd. 

Strongly Strongly 

Agree Agree Disagree Disagree 

9. The scoring rubric has the right number of performance levels (It currently has 
three — Basic, Proficient and Advanced). 

Yes No 

10. If you answered “No” to question 9, how many performance levels should the 
rubric have? 

2 4 5 6 

1 1 . Overall, the time and effort the school invested in scoring/benchmarking was 
worthwhile. 

Strongly Strongly 

Agree Agree Disagree Disagree 



Suggestions for improving portfolio scoring/benchmarking: 



THANK YOU FOR SCORING/BENCHMARKING!!!!! 
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Drake High School 
1999 

Professional Skills Assessment Portfolio 
Score Sheet 



student Name 



Scoring procedure 

1 . Read the scoring guide on the back of this sheet 

2. Read the portfolio, looking for evidence described in the scoring guide 

3. Highlight the scoring guide where the description matches the evidence in the 
portfolio 

4. Assign a score for each dimension 

5. Assign an overall score 




1^0 



Logged 
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